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SYNOPSIS 

The  objective  of  this  study  was  the  development  and  validation  of 
an  effective  method  for  obtaining  post  formal  training  feedback  information 
suitable  for  use  throughout  the  Navy  training  system.  The  continuous 
collection  of  this  information  from  operational  units  concerning  the  job 
performance  of  school  graduates  is  vital  for  maintaining  up-to-date, 
effective  training  programs.  Although  a number  of  unrelated  approaches 
for  obtaining  post  formal  training  feedback  exist  in  different  Navy 
schools,  there  is  no  standard,  systematic  method  for  obtaining  such 
data.  To  fill  this  gap  in  the  evaluation  of  training  programs,  the 
Chief  of  Naval  Education  and  Training  (CNET),  N-34,  assigned  to  the 
Training  Analysis  and  Evaluation  Group  (TAEG)  the  task  of  developing 
such  a method. 

A preliminary  examination  of  existing  feedback  techniques  indicated 
that  the  mail -out  questionnaire  and  the  personal  interview  were  the 
general  approaches  most  suitable  for  this  purpose.  The  questionnaire 
method  has  the  potential  for  inexpensively  providing  reliable  data  from 
the  widest  possible  range  of  operational  units.  However,  low  return 
rates  and  data  of  poor  quality  have  been  the  more  typical  results  when 
this  feedback  method  has  been  used  by  Navy  schools.  Prior  to  selection 
and  development  of  a feedback  method  for  Navy-wide  use,  it  was  essential 
to  resolve  many  questions  about  the  effectiveness  of  mail -out  feedback 
methods  and  to  compare  their  performance  with  that  of  more  highly  esteemed 
personal  interviews.  To  do  this,  an  empirical  comparison  of  mail -out 
and  personal  interview  feedback  methods  was  undertaken  in  a military 
training  setting  where  many  training  problems  had  already  been  identified. 
Different  feedback  methods  could  thus  be  easily  compared  on  the  accuracy 
with  which  they  disclosed  these  known  training  problems. 

The  Radioman  "A"  School  was  selected  for  this  development  and 
comparison  of  feedback  methods.  The  school  had  recently  undergone  a 
major  curriculum  revision  based  on  a job  task  analysis.  The  new  curricu- 
lum had  not  yet  been  implemented  and  the  many  training  problems  identified 
in  the  old  curriculum  would  serve  to  measure  the  effectiveness  of  the 
different  feedback  methods.  In  addition,  the  large  number  of  graduates 
produced  by  the  school  provided  the  study  team  with  the  opportunity  to 
test  a number  of  different  mail-out  instruments  and  administration 
techniques. 

Questionnaires  were  mailed  to  996  recent  Radioman  "A"  School  graduates 
and  590  supervisors  of  such  graduates.  The  names  and  locations  of  these 
personnel  had  been  confirmed  by  letters  returned  from  the  operational 
units  prior  to  questionnaire  mailing.  Three  different  questionnaire 
formats  were  evaluated.  The  first  was  a long  form  (134  items)  made  up 
of  specific  training  and  job  task  statements.  The  second  was  a short 
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form  (15  items)  made  from  a list  of  general  job  task  statements  that 
subsumed  the  specific  job  tasks  of  the  long  form.  On  both  forms,  the 
respondent  was  asked  to  rate  (1)  the  frequency  with  which  the  trainee 
performed  the  task,  (2)  the  criticality  of  the  task,  and  (3)  how  well 
the  trainee  could  perform  the  task  upon  arrival  at  the  operational  unit. 

The  third  instrument  was  a card-sorting  technique  in  which  the  134  items 
from  the  long  questionnaire  were  printed  on  small  cards  to  be  sorted  by 
the  respondent  into  categories  related  to  task  frequency  and  to  the 
trainee's  ability  to  perform  the  task.  In  addition  to  mail-out  procedures, 
a group  of  59  trainees  and  37  supervisors  were  interviewed  face-to-face 
using  a structured  interview  based  on  the  long  questionnaire. 

The  data  obtained  by  each  method  were  analyzed  for  the  accuracy  and 
comprehensiveness  with  which  the  rating  scale  data  identified  the  known 
training  problems  that  had  existed  for  the  graduates.  Return  rate 
statistics  were  also  computed  for  the  different  mail -out  instruments  as 
well  as  statistics  on  time  until  instrument  return.  Separate  analyses 
on  these  variables  were  conducted  for  (1)  trainees  and  supervisors,  (2) 
persons  who  expected  the  questionnaire  and  persons  who  did  not,  (3) 
graduates  with  different  times  from  graduation  prior  to  evaluation,  (4) 
graduates  with  different  class  standings,  and  (5)  graduates  with  different 
duty  stations.  The  last  variable  was  investigated  because  many  of  the 
problems  of  the  old  Radioman  "A"  School  curriculum  were  related  to  the 
inappropriate  training  on  shipboard  systems  given  to  persons  assigned  to 
shore  stations. 

All  of  the  instruments  and  procedures  included  in  the  study  were  at 
least  moderately  successful  in  identifying  the  known  training  problems 
as  well  as  identifying  successful  training.  The  long  questionnaire 
based  on  specific  job  task  statements  was  particularly  effective  and 
provided  data  that  were  nearly  identical  to  the  data  obtained  from  the 
personal  interview  method.  These  data  were  highly  reliable  (i.e., 
consistent  from  one  respondent  to  another),  and  they  accurately  and 
comprehensively  identified  the  shortcomings  known  to  exist  in  the  old 
Radioman  "A"  School  curriculum.  Supervisory  personnel  provided  somewhat 
better  quality  information  than  trainees,  although  both  were  excellent 
sources  of  data  about  training  problems  and  both  should  be  called  on  to 
provide  feedback  information. 

The  overall  rate  of  return  for  instruments  was  satisfactory  to  meet 
training  needs.  The  rate  of  return  was  59  percent  for  supervisors  and 
31  percent  for  trainees.  The  return  rate  for  short  form  questionnaires 
was  significantly  greater  than  for  the  long  form  with  most  of  the  differ- 
ence between  the  instruments  contributed  by  the  trainees.  Despite  a 
higher  rate  of  return,  the  short  form  failed  to  identify  many  specific 
training  problems  that  were  disclosed  by  the  long  questionnaire. 
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An  expectation  or  "set"  to  receive  the  mail -out  instrument  was 
found  to  significantly  improve  the  return  rate  of  trainees.  Such  a 
"set"  should  be  established  in  the  graduate,  either  as  it  was  done  in 
this  study  by  preceding  the  questionnaire  with  a letter  or  by  contacting 
him  prior  to  his  departure  from  school.  The  optimal  time  lapse  between 
graduation  and  administration  of  the  feedback  instrument  was  determined 
to  be  five  to  seven  months. 

Because  of  the  excellent  results  obtained  in  this  study  with  the 
questionnaire  based  on  specific  job  tasks,  it  is  recommended  that  the 
CNET  adopt  this  particular  instrument  format  and  associated  procedures 
as  the  major  method  for  obtaining  post  formal  training  feedback  information 
in  Navy  schools.  The  recommended  feedback/curriculum  revision  cycle 
takes  18  months  plus  the  duration  of  the  course.  A minimum  number  of 
personnel  are  required  to  develop  and  administer  the  instruments  and  to 
analyze  the  data.  Procedures  are  straightforward  and  require  no  particular 
training  of  the  person  or  persons  assigned  to  carry  them  out.  These 
procedures  are  also  described  in  a companion  report  (TAEG  "Feedback 
Manual,"  June  1975)  written  specifically  for  the  training  personnel  who 
will  be  primarily  responsible  for  implementing  them. 
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SECTION  I 
INTRODUCTION 

A training  program  is  effective  only  to  the  extent  that  it  meets 
the  needs  of  the  Fleet  and  other  operational  units  for  qualified  personnel. 
These  needs  must  be  given  the  highest  priority  in  the  design  of  training 
programs  and  also  in  the  assessment  of  these  programs.  Course  design 
based  on  job  task  analysis  meets  this  criterion,  and  procedures  already 
exist  for  this  purpose  (Butler,  1972;  Branson,  1974;  Rundquist,  1970; 

Smith,  1971).  However  there  is  a gap  in  the  evaluation  of  training 
programs  since  standard  and  systematic  procedures  do  not  exist  for 
providing  schools  with  job  performance  data  on  their  graduates.  Such 
data  is  needed  (1)  to  correct  the  errors  that  inevitably  result  even 
with  proper  course  design  and  (2)  to  update  courses  in  the  face  of 
changing  job  requirements.  To  fill  this  gap,  the  Chief  of  Naval  Education 
and  Training  (CNET),  N-34,  assigned  to  the  Training  Analysis  and  Evaluation 
Group  (TAEG)  the  task  of  developing  a feedback  method  which  would  system- 
atically and  cost  effectively  provide  Navy  schools  with  critical  informa- 
tion from  operational  units  about  training  outcomes.  To  achieve  this 
task,  an  extensive  study  of  training  feedback  methodology  was  conducted 
by  a three-person  team  during  the  period  from  November  1973  to  March 
1975. 

OBJECTIVES  OF  THE  STUDY 

This  study  was  undertaken  to  accomplish  the  following  objectives: 

1.  To  determine  the  utility  of  existing  training  feedback  methods, 
particularly  those  which  utilize  inexpensive  mail-out  procedures. 

2.  To  develop  an  instrument  for  Navy  training  feedback,  procedures 
for  its  administration,  and  procedures  for  utilization  of  the  data  it 
provides. 

3.  To  establish  the  effectiveness  and  cost  of  the  selected  feedback 
method. 

4.  To  provide  a manual  that  will  enable  relatively  unsophisticated 
personnel  in  Navy  schools  to  construct  and  administer  the  instrument  and 
to  use  the  data  provided. 

BACKGROUND 

Much  of  the  impetus  for  this  project  stemmed  from  a study  by  Peters 
and  Chambers  (1964)  of  Navy  training  feedback  methods  and  requirements. 

The  report  identified  many  problems  that  existed  in  the  evaluation  of 
Navy  training  programs  ranging  from  the  general  lack  of  awareness  of  the 
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importance  of  training  feedback  to  the  absence  of  specific  forms  and 
procedures  for  obtaining  feedback  information.  A series  of  recommenda- 
tions was  provided  for  improvement  of  the  Navy  training  feedback  process. 

Bildhauer  (1971)  noted  that  much  the  same  situation  still  existed 
in  evaluation  of  Navy  training  programs  as  was  documented  in  the  earlier 
Peters  and  Chambers  (1964)  report.  He  concluded,  "the  environment  of 
most  training  programs  actually  prevents  the  accomplishment  of  feedback." 

To  aid  in  correcting  this  situation,  Bildhauer  developed  plans,  procedures 
and  an  organizational  structure  for  an  integrated  Navy-wide  Training 
Feedback  Subsystem  (TFS). 

Although  the  Bildhauer  (1971)  report  was  prepared  for  the  Chief  of 
Naval  Personnel,  it  became  an  important  working  paper  in  the  Training 
Appraisal  Branch  (N-34)  of  the  new  CNET.  Development  of  the  TFS  continued 
in  this  branch  including  a study  by  Lane  (1972)  of  the  use  of  question- 
naire methods  for  obtaining  field  evaluation  data.  The  conclusion  of 
that  report  was  that  questionnaires  were  suitable  for  this  purpose,  if 
proper  procedures  for  their  construction  and  administration  were  used. 
Another  result  of  TFS  development  at  CNET  was  a second  report  by  Bildhauer 
(1973)  which  documented  the  evolution  of  the  TFS  from  his  earlier  concep- 
tion and  described  the  development  and  organizational  structure  of  one 
local  evaluation  unit  (Service  School  Conmand,  Orlando).  From  the 
standpoint  of  the  present  study,  the  most  critical  outcome  of  CNET  work 
on  the  TFS  was  the  assignment  to  TAEG  of  the  task  of  developing  a vehicle 
to  provide  feedback  information  from  the  operational  units  to  the  schools. 

CHARACTERISTICS  OF  AN  EFFECTIVE  FEEDBACK  METHOD 

An  optimally  effective  post  formal  training  feedback  method  would 
provide  accurate  information  to  the  schools  related  to  the  success  or 
failure  of  school  training  for  every  job  task  that  the  trainee  may  be 
asked  to  perform.  The  training  failures  which  such  a feedback  method 
would  help  to  identify  fall  into  the  categories  of  undertraining  (including 
missing  training),  overtraining,  and  irrelevant  training. 

Undertraining  is  a failure  of  the  school  to  provide  sufficient 
skills  to  the  trainee  to  enable  him  to  adequately  perform  a job  task. 

It  is  the  most  critical  training  failure  since  it  reduces  the  operational 
readiness  of  the  unit  to  which  the  trainee  is  assigned.  Overtraining 
occurs  when  more  training  is  given  than  is  needed  to  perform  a job  task 
and  training  resources  are  thereby  wasted.  An  even  larger  waste  of 
resources  occurs  in  irrelevant  training,  where  skills  and  knowledge  are 
acquired  that  have  no  bearing  on  present  or  future  job  performance. 

Any  post  formal  training  feedback  method  that  provides  accurate 
data  on  the  existence  of  each  of  these  training  failures,  as  well  as 
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identifying  training  successes,  will  necessarily  be  reliable  and  valid. 
However,  a number  of  additional  desirable  characteristics  exist  for  an 
effective  feedback  method.  Probably  the  most  important  of  these  is  low 
cost  for  development  and  administration.  Some  methods,  such  as  personal 
interviews  with  Fleet  personnel,  become  extremely  expensive  if  enough 
visits  are  made  to  provide  reliable  data,  and  mail -out  questionnaires 
have  been  found  to  be  much  less  expensive  (Scott,  1961).  It  is  also 
desirable  that  a training  feedback  method  operates  quickly.  Training 
problems  should  be  corrected  as  soon  as  possible  following  their  appear- 
ance and,  therefore,  the  time  lag  to  obtain  indications  of  these  problems 
must  also  be  short.  In  addition  to  low  cost  and  speed,  it  is  important 
that  the  method  be  simple  and  easy  to  administer  since  any  procedure 
that  depends  on  complicated  analysis  techniques  will  probably  not  be 
carried  out.  Finally,  the  training  feedback  method  should  require  a 
minimum  number  of  personnel  for  its  execution.  Military  training  units 
are  often  understaffed  and  a feedback  procedure  that  requires  a great 
amount  of  time  by  training  personnel  for  its  administration  will  probably 
be  neglected. 

POST  FORMAL  TRAINING  FEEDBACK  TECHNIQUES 

The  ultimate  source  of  feedback  information  is  the  performance  of 
the  trainee  on  the  job,  and  many  ways  exist  to  obtain  such  data.  Probably 
no  training  program  is  completely  lacking  in  procedures  for  obtaining 
such  information.  However,  not  all  methods  have  the  capability  of 
identifying  all  the  classes  of  training  problems  discussed  above.  For 
example,  analysis  of  records  of  equipment  failure  can  provide  information 
about  undertraining,  but  little  or  no  information  about  overtraining  and 
irrelevant  training.  To  a lesser  extent,  proficiency  testing  shares 
this  weakness.  Techniques,  such  as  questionnaires,  which  do  have  the 
potential  to  provide  data  on  all  three  classes  of  training  failures, 
often  fail  in  the  attempt,  because  low  return  rates  or  poor  question- 
naire design  reduce  the  reliability  of  the  data.  Even  when  usable  data 
are  obtained  in  sufficient  quantities,  it  often  is  not  properly  analyzed 
to  identify  significant  trends. 

The  existing  techniques  were  the  starting  point  in  our  search  for 
an  effective  Navy  post  formal  training  feedback  method.  They  are  presented 
below  to  provide  an  overview  of  possible  procedures  and  their  relative 
adequacy.  If  time  and  personnel  are  available,  some  of  these  methods 
can  be  used  as  sources  of  feedback  data  along  with  the  particular  long 
questionnaire  method  recommended  in  this  report. 

PROFICIENCY  TESTING.  Objective  criteria  which  indicate  the  success  of 
the  trainee's  performance  can  be  measured.  These  include  such  things  as 
meter  readings,  signal-to-noise  ratios,  and  amplification  factors. 

Along  with  these  products  of  performance,  the  time  to  complete  tasks  and 
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the  manner  of  their  completion  can  be  observed.  These  various  objective 
measurements  can  then  be  compared  to  standards  that  have  been  established 
as  representing  proper  performance.  Since  many  aspects  of  military  jobs 
are  not  routinely  performed,  simulated  environments  can  sometimes  be 
utilized  to  allow  objective  indications  of  performance  for  these  job 
behaviors.  Although  the  resultant  data  provide  excellent  feedback 
information  in  most  situations,  the  disadvantages  are  likely  to  outweigh 
the  benefits  obtained.  This  method  is  costly,  time  consuming  and  requires 
highly  skilled  personnel  to  develop  and  administer  the  measuring  instruments. 
In  addition,  the  equipment  may  not  be  available  for  testing  since  its 
use  for  this  purpose  could  preclude  its  use  in  day-to-day  operations. 

For  more  detailed  information  on  the  reasons  for  avoiding  performance 
testing  as  a feedback  technique  see  Harris  and  Mackie  (1962). 

JOB-RELEVANT  KNOWLEDGE  TESTING.  In  addition  to  measurement  of  performance 
in  real  or  simulated  environments,  hypothetical  job  situations  can  be 
established  by  the  use  of  written  questions  to  determine  if  the  knowledge 
required  for  successful  performance  is  present  in  the  job  incumbent. 

This  is  basically  the  approach  of  the  Personnel  Testing  and  Evaluation 
Program  (PTEP)  being  used  by  the  Fleet  Guided  Missile  School.  This 
procedure  is  closely  related  to  proficiency  testing  and  promises  to  have 
value  for  the  training  community.  The  PTEP  is  presently  being  used  by 
the  subsurface  community  for  identification  of  specific  personnel  perform- 
ance weaknesses,  but  its  potential  as  a training  feedback  technique 
should  be  explored. 

ANALYSIS  OF  EXISTING  RECORDS.  Data  on  equipment  failures  from  records 
such  as  3M,  Maintenance  Data  Forms,  Records  of  Mobile  Technical  Units, 

Fleet  Training  Group  Reports,  are  primarily  hardware  oriented,  but  to 
the  extent  that  they  are  indices  of  student  performance  (failures),  they 
are  a source  of  training  feedback  information.  Records  such  as  Safety 
Reports  and  Reports  of  Refresher  Training  can  also  give  insight  into 
training  problems  to  the  extent  that  they  reflect  poor  performance  of 
the  trainee  on  the  job.  One  problem  with  this  method  of  obtaining 
information  is  that  these  records  are  not  widely  distributed.  Additionally, 
the  data  must  be  verified  by  cross-checking  other  sources. 

VISITS:  INFORMAL/UNSTRUCTURED.  Training  personnel  may  visit  the  job 

site  for  general  discussion  of  trainee  performance  rather  than  to  obtain 
data  on  a specific  training  issue.  This  approach  suffers  because  of  the 
small  amount  of  information  obtained.  It  may  only  have  the  advantage  of 
demonstrating  to  Fleet  units  that  training  personnel  care  enough  about 
supporting  them  to  come  aboard.  These  visits  can  be  made  more  profitable 
by  adding  structure  to  the  visit  through  the  use  of  proper  questioning 
techniques  (see  structured  interview  below).  Considering  the  value  of 
the  data  usually  collected,  the  cost  of  this  method  makes  it  a question- 
able technique  for  continuing  use. 
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INFORMAL  FEEDBACK.  At  times  people  in  the  operational  environment  wish 
to  express  their  concern  for,  or  satisfaction  with,  training.  Such 
unsolicited  information  may  be  transmitted  by  letter,  phone  call,  or 
personal  contact.  Much  of  this  information  is  broad  and  may  lack  impor- 
tant details.  This  data  should  be  considered  as  a starting  point  for 
further  investigation,  rather  than  taken  directly  at  face  value. 

ROTATION  OF  FLEET/TRAINING  PERSONNEL.  Much  information  about  the  adequacy 
of  training  may  be  obtained  from  newly  reporting  school  personnel.  They 
are  usually  capable,  well -motivated  people  with  knowledge  about  how 
effectively  school  graduates  perform  their  jobs.  Information  should  be 
obtained  during  skilled  debriefing  sessions  shortly  after  their  arrival. 
All  opinions  should  be  backed  by  observation  and  preferably  with  specific 
examples.  It  is  essential  that  the  information  obtained  in  this  way  be 
evaluated  in  the  light  of  the  respondent's  experience  and  qualifications. 
Cross-checking  with  the  operational  unit  on  specific  training  failures 
would  be  important  for  validating  the  information  received.  The  use  of 
a questionnaire  which  provides  for  ratings  of  performance  on  all  job 
tasks  should  be  used  for  debriefing  these  personnel. 

PERFORMANCE  DIARY.  The  Performance  Diary  is  a running  report,  by  the 
school  graduate,  of  the  tasks  he  performs  and  the  problems  he  encounters 
on  the  job.  It  demands  a high  degree  of  cooperation  between  the  school 
and  the  job  incumbent  and  requires  much  time  and  effort  on  the  part  of 
the  trainee.  When  suitable  personnel  have  been  found  to  keep  such 
diaries,  the  information  contained  therein  has  been  useful  for  training 
purposes.  However,  the  data  obtained  must  be  carefully  evaluated.  The 
fact  that  a capable  technician  feels  a particular  portion  of  the  training 
was  helpful  to  him,  does  not  mean  that  it  would  help  the  average  student. 
On  the  other  hand,  if  the  diarist  reports  that  he  has  difficulty,  or 
cannot  perform  an  operation  which  was  taught  in  school,  an  examination 
of  that  phase  of  instruction  may  be  in  order. 

VISITS:  STRUCTURED  INTERVIEW.  The  details  of  the  job  are  structured  in 

terms  of  statements  derived  from  a job  task  analysis.  Interview  forms 
(checklists,  rating  scales,  questionnaires,  etc.)  are  constructed  which 
can  be  completed  within  one  to  three  hours  of  interview  time.  Training 
personnel  visit  the  operational  unit  and  administer  the  forms  on  a face- 
to-face  basis.  This  technique  has  been  used  satisfactorily  by  Bil inski 
and  Saylor  (1972)  and  Standlee,  et  al . , (1972)  who  examined  the  job 
performance  of  Navy  Storekeeper  graduates  and  electronic  maintenance 
personnel,  respectively.  Because  of  its  structure,  this  approach  yields 
more  information  (and  more  reliable  information)  than  the  unstructured 
interview.  It  is  superior  to  mail-out  procedures  in  that  there  is  no 
problem  about  return  rate  or  waiting  for  replies.  Also,  much  additional 
information  can  be  gained  about  frequency  of  operations  and  maintenance 
as  well  as  about  tasks  performed  other  than  those  pertaining  to  the 
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technical  rating.  While  this  technique  s usually  too  costly  in  time 
and  money  to  be  used  as  a continuing  feedback  method,  it  is  valuable  for 
use  in  locations  where  operational  units  are  close  to  the  school. 

MAIL-OUT  QUESTIONNAIRE.  Questionnaires  have  been  used  frequently  to 
obtain  training  feedback  data.  They  offer  the  advantage  of  being  less 
costly  than  the  personal  interview  method.  In  addition,  the  question- 
naire is  relatively  free  of  interviewer  bias.  The  format  for  construc- 
tion of  questionnaires  has  varied  to  include  open-ended  questions, 
closed-ended  questions,  and/or  rating  scales.  Items  have  been  based  on 
task  statements  obtained  .through  job  task  analysis  (as  in  this  study); 
on  job  dimensions  determined  by  factor  analysis,  as  in  Siegel  and 
Federman's  (1970)  study  of  the  electronics  ratings;  or,  as  is  most  often 
the  case,  on  the  existing  school  curriculum.  Procedures  for  administra- 
tion have  also  taken  many  different  forms.  The  questionnaire  has  been 
carried  by  the  graduate  to  his  duty  station  or  mailed  at  some  specified 
time  after  graduation.  They  have  been  administered  to  the  trainee,  to 
the  supervisor,  or  to  both.  Also,  different  amounts  of  time  have  been 
permitted  to  lapse  following  graduation  prior  to  mailing  of  the  question- 
naire. Navy  use  of  this  technique  has  suffered  from  a low  return  rate. 

An  average  return  rate  of  only  10  percent  was  reported  in  a recent 
survey  of  Navy  feedback  methods  (TAEG  Report  Mo.  12-1,  1973).  Another 
disadvantage  often  mentioned  has  been  the  lack  of  reliability  of  this 
data.  It  has  been  argued,  however,  that  satisfactory  return  rates  and 
useful  reliable  data  can  be  obtained  when  appropriate  procedures  are 
followed  in  their  construction  and  administration  (Lane,  1972). 

FACTORS  THAT  IMPACT  ON  TRAINING  FEEDBACK 

Many  factors  contribute  to  the  widely  varying  effectiveness  observed 
in  training  feedback  efforts.  For  example,  the  very  poor  results  noted 
in  the  recent  survey  of  Navy  feedback  efforts  (TAEG  Report  No.  12-1, 

1973)  were  attributed  largely  to  the  predominant  use  of  mail -out  instru- 
ments instead  of  personal  interviews.  Other  factors  that  are,  or  may 
be,  related  to  the  success  or  failure  of  feedback  efforts,  particularly 
those  employing  questionnaires,  include  the  length  and  format  of  question- 
naires, the  person  or  persons  providing  the  information,  the  time  lapse 
following  graduation  prior  to  gathering  the  data,  the  ability  and 
conscientiousness  of  the  person  providing  feedback  data,  and  the  effect 
of  discussion  or  notification  of  the  feedback  effort  prior  to  actual 
administration.  Each  of  these  factors  is  discussed  below.  Particular 
attention  is  given  to  the  need  to  include  each  factor  in  an  empirical 
test  of  feedback  methods  to  clarify  the  role  of  the  factor  in  training 
feedback  effectiveness. 
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PERSONAL  VS.  MAIL-OUT  CONTACT.  Many  studies  indicate  that  well -constructed 
and  carefully  administered  mail -out  instruments  can  provide  accurate  and 
reliable  data.  These  include  studies  (Bougler,  1970;  Scott,  1961)  in 
which  no  differences  were  found  between  data  obtained  from  mail -out 
interviews  and  personal  interviews.  However,  mail -out  procedures  were 
severely  criticized  in  the  recent  survey  of  training  feedback  practices 
in  the  Navy  (TAEG  Report  No.  12-1,  1973)  because  of  the  low  return  rate 
and  because  the  data  were  often  judged  to  be  unreliable.  Prior  to  the 
selection  and  development  of  a feedback  method  for  Navy-wide  use,  it  was 
very  important  to  resolve  these  contradictory  positions  about  the  effec- 
tiveness of  mail-out  procedures.  Since  a comparison  of  mail-out  interviews 
and  personal  interviews  had  not  been  done  in  the  area  of  military  training, 
an  experiment  was  needed  to  compare  these  procedures  in  a Navy  training 
setting.  If  mail-out  procedures  were  found  adequate  for  the  job,  the 
savings  for  the  training  community  would  be  huge. 

LENGTH  AND  FORMAT  OF  FEEDBACK  INSTRUMENTS.  Long  questionnaires  provide 
the  opportunity  for  a more  comprehensive  evaluation  of  training  than 
short  questionnaires.  However,  long  questionnaires  are  more  apt  to 
produce  fatigue  and  boredom  in  the  respondent.  Thus,  the  benefits  of 
length  may  be  cancelled  because  of  reduced  accuracy  of  the  data  or 
because  very  few  questionnaires  are  returned.  For  these  reasons,  short 
questionnaires  might  provide  better  data  than  long  questionnaires  despite 
their  limitations.  To  obtain  better  knowledge  of  the  relationship 
between  questionnaire  length  and  effectiveness,  it  was  essential  to 
include  both  long  and  short  questionnaires  in  the  present  experiment 
comparing  mail -out  interviews  with  personal  interviews.  Data  from  the 
long  questionnaires  would  also  provide  the  opportunity  to  assess  the 
accuracy  of  questionnaire  data  from  different  item  locations.  Presumably, 
items  appearing  early  would  provide  better  data  and  the  number  of  items 
into  the  questionnaire  at  which  data  become  less  accurate  would  provide 
an  indication  of  the  maximum  length  for  questionnaires. 

Responding  to  mail-out  questionnaires  is  closely  related  to  the 
kind  of  behavior  required  for  paper  and  pencil  examinations.  Some  of 
the  poor  results  observed  in  Navy  use  of  feedback  questionnaires  could 
be  related  to  an  aversion  for  such  examination-like  tasks.  Bil inski  and 
Saylor  (1972)  used  a novel  format  for  feedback  instruments  in  an  assess- 
ment of  training  for  the  Navy  Storekeeper  rating.  They  conducted 
structured  interviews  in  which  cards,  with  printed  task  statements,  were 
sorted  into  one  of  several  different  categories  related  to  levels  of 
performance  of  a trainee  on  the  task.  This  successful  format  and  proce- 
dure for  interviews  appeared  to  be  adaptable  for  mail -out  instruments. 
Because  of  its  simplicity  and  novelty,  it  might  provide  a higher  return 
rate  and  more  reliable  and  more  valid  data  than  more  traditional  formats. 
The  possible  advantages  of  this  technique  argued  for  its  inclusion  in 
the  study  comparing  feedback  methods. 
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TRAINEE  VS.  SUPERVISOR  ADMINISTRATION  PROCEDURE.  Probably  the  most 
important  question  related  to  administration  of  any  feedback  method  is 
that  of  who  should  provide  the  ratings  of  trainee  performance.  The 
trainee  has  a direct  and  recent  knowledge  of  school  training  and  also 
has  a direct,  though  limited,  acquaintance  with  the  requirements  of  his 
new  job.  On  the  other  hand,  the  supervisor  has  a much  more  complete  and 
accurate  conception  of  adequate  job  performance  but  little  knowledge  of 
current  training  at  the  school.  Thus  both  the  trainee  and  supervisor 
have  special  perspectives  that,  in  theory,  make  input  from  both  essential 
for  a complete  training  feedback  data  base.  However,  even  if  their  data 
were  identical,  it  would  be  useful  to  share  the  workload  between  trainees 
and  supervisors.  If  both  trainees  and  supervisors  are  to  be  contacted, 
the  additional  question  arises  as  to  whether  both  members  of  a trainee- 
supervisor  pair  should  be  interrogated  or  only  one  member  from  each 
pair.  An  experiment  investigating  different  methods  could  also  be  used 
to  compare  the  accuracy  and  reliability  of  data  from  these  different 
groups  of  respondents. 

TIME  LAPSE  PRIOR  TO  ADMINISTRATION.  Another  factor  impacting  on  training 
feedback  is  the  lapse  of  time  following  graduation  prior  to  mailing  a 
questionnaire  or  conducting  an  interview.  More  time  on  the  job  will 
allow  a better  conception  of  job  requirements  to  develop  in  the  trainee, 
but  it  will  also  lead  to  more  forgetting  of  school  training  problems, 
particularly  those  problems  that  are  corrected  by  on-the-job  training. 

A similar  benefit  and  disadvantage  would  accrue  to  supervisors  as  elapsed 
time  between  graduation  and  evaluation  increases.  They  would  have  more 
time  to  observe  the  trainee  on  the  job  but  more  possibility  would  exist 
for  confounding  of  on-the-job  and  school  training.  In  a study  of  feedback 
methods,  the  data  from  respondent  groups  made  up  of  persons  with  different 
times  from  graduation  could  be  compared  to  help  provide  an  answer  to  the 
complex  question  regarding  the  optimal  time  from  graduation  to  wait 
prior  to  obtaining  training  feedback. 

EXPECTATION  OR  "SET"  TO  RESPOND.  Representations  of  the  future  such  as 
goals  and  expectations  play  an  extremely  important  role  in  human  behavior. 
A person  who  receives  notification  that  he  will  be  receiving  a feedback 
questionnaire  will  probably  be  more  receptive  to  it  when  it  arrives  than 
the  person  who  is  surprised  by  its  appearance.  However,  such  expectations 
might  work  to  reduce  the  accuracy  and  care  of  responding,  at  least  for 
some  respondent  groups.  A test  of  the  effects  of  prior  notification  by 
"expectation  letters"  was  needed. 

CLASS  STANDING  OF  THE  GRADUATE.  Training  is  probably  experienced  very 
differently  by  persons  who  graduate  at  the  top,  middle  and  bottom  of 
their  classes.  In  addition,  the  more  conscientious  and  able  student  may 
be  a more  conscientious  and  accurate  provider  of  feedback  data.  Good 
reasons  thus  exist  for  analyzing  feedback  data  separately  for  these 
groups  and  an  examination  of  these  possible  differences  was  included  in 
the  study  of  feedback  methods. 
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SECTION  II 
TECHNICAL  APPROACH 

A number  of  issues  critical  to  the  selection  and  development  of  a 
Navy  training  feedback  method  required  empirical  data  for  resolution. 

For  this  reason,  an  investigation  was  conducted  in  which  different 
feedback  methods  were  applied  in  an  actual  Navy  school /job  setting. 
Specific  training  problems  were  already  known  to  exist  at  the  chosen 
test  location  (the  Radioman  "A"  School)  and  this  allowed  comparison  of 
the  different  instruments  and  procedures  to  determine  the  effectiveness 
of  each  for  such  problem  identification.  Of  more  importance,  it  provided 
the  answer  to  the  question  of  whether  or  not  any  of  the  methods  could 
provide  this  information  from  the  operational  units  to  schools  about 
training  problems. 

VARIABLES  INVESTIGATED  IN  THE  STUDY 

Each  of  the  factors  impacting  on  feedback  effectiveness  described 
in  the  previous  section  became  a variable  that  was  investigated  in  the 
study.  In  addition,  a variable  based  on  where  the  graduate  was  stationed 
was  included,  since  this  was  closely  related  to  the  problems  known  to 
exist  in  the  training  curriculum  for  Radioman  "A"  School. 

The  following  variables  were  investigated  and  are  discussed  below. 
Variables  related  to  the  instruments  included: 

Type  of  instrument 

Normal  vs.  reversed  item  order 

Variables  related  to  characteristics  of  personnel  receiving  instruments 
included: 


Supervisor  or  trainee  respondent 
Joint  vs.  single  administration 
Class  standing  of  graduate 
Time  from  graduation 
. Expectation  vs.  non-expectation 
Ship  vs.  shore  duty  station 

The  major  variable  under  investigation  was  the  type  of  feedback 
instrument.  Three  instruments  used  mail-out  procedures  to  obtain  data 
and  one  used  a structured  personal  interview.  One  mail -out  instrument 
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was  a long  questionnaire  that  required  ratings  of  task  frequency,  task 
criticality,  and  trainee  performance  on  134  specific  job  tasks  performed 
by  Radiomen.  Another  was  a short  questionnaire  that  required  the  same 
ratings  on  15  general  job  tasks  which  subsumed  the  specific  job  tasks  of 
the  long  questionnaire.  The  final  mail-out  instrument  was  a card-sort 
procedure  where  134  cards,  with  the  specific  job  tasks,  were  sorted  into 
different  stacks  related  to  the  frequency  of  task  performance  and  the 
level  of  trainee  performance  on  the  task.  The  structured  interview  used 
the  long  questionnaire,  but  it  was  administered  in  a face-to-face  situation 
with  the  interviewer  writing  down  the  ratings  as  they  were  told  to  him 
by  the  person  being  interviewed. 

A second  variable  in  the  study  also  involved  the  type  of  instrument 
used.  This  was  a two-level  variable  defined  by  whether  the  items  on  the 
instrument  were  in  normal  or  reversed  order.  This  variable  was  included 
to  obtain  information  about  possible  differences  in  ratings  of  an  item 
depending  upon  the  position  of  the  item  in  the  questionnaire.  It  was 
completely  "crossed"  with  the  three-level  factor  of  mail -out  instrument 
type.  This  means  that  one-half  of  the  questionnaires  of  each  type  had 
reversed  item  order  and  one-half  had  normal  item  order.  It  also  means 
that  one-third  of  the  reversed- item-order  instruments  were  long  question- 
naires, one-third  were  short  questionnaires,  and  one-third  were  card 
sort  instruments.  Finally,  this  complete  "crossing"  of  these  two  factors 
implies  that  the  same  proportions  (one-third  of  each  questionnaire  type) 
held  for  the  instruments  with  normal -item-order. 

The  other  six  variables  included  in  the  investigation  are  all 
related  to  characteristics  of  the  personnel  who  were  mailed  the  feedback 
instruments.  The  most  important  of  these  variables  involved  whether  the 
respondent  was  a supervisor  or  trainee.  Another  two-level  variable 
(referred  to  as  joint  vs.  single  administration)  was  determined  by 
whether  the  instrument  was  mailed  to  only  one  member  of  the  supervisor- 
trainee  pair  or  to  both  members  of  the  pair.  A three- level  variable  was 
defined  by  the  class  standing  of  the  graduate.  Class  graduation  rosters 
were  divided  into  thirds  on  the  basis  of  class  standing  to  allow  separate 
investigation  of  feedback  accuracy  for  groups  at  the  top,  middle  and 
bottom  of  their  classes.  A four-level  variable  was  determined  by  the 
time  lapse  from  graduation  prior  to  administration  of  the  instrument. 

The  groups  were  4 to  6 months  time  lapse,  7 to  9 months,  10  to  12  months, 
and  greater  than  12  months.  A two-level  variable  was  determined  by 
whether  or  not  the  person  had  received  an  "expectation  letter"  prior  to 
the  questionnaire,  informing  him  of  its  imminent  arrival.  The  final 
two-level  variable  was  related  to  whether  the  graduate  was  assigned  to  a 
ship  or  to  a shore  duty  station. 
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Some  of  these  variables,  such  as  duty  station  of  the  graduate  and 
time  from  graduation,  were  not  under  the  control  of  the  investigators. 
Others,  such  as  questionnaire  type  and  the  presence  or  absence  of 
"expectation  letters,"  were  assigned  randomly  to  trainees  and  supervisors. 
Wherever  possible,  equal  numbers  of  each  level  of  a variable  were  included 
in  every  level  of  every  other  variable.  For  example,  equal  numbers  of 
the  three  different  mail -out  instruments  were  assigned  to  personnel 
stationed  at  shore  duty  stations  and  to  personnel  aboard  ships.  This 
procedure  allows  a single  experiment  to  provide  accurate  data  on  each  of 
the  factors  which  is  included  and  "crossed"  with  other  factors.  Eight 
single-factor  experiments  would  be  required  to  provide  the  data  available 
from  the  present  study  with  its  eight  different  factors.  In  addition, 
this  multi -factor  approach  allows  a finer  analysis  of  the  data  in  terms 
of  sub-groups  of  persons.  Since  a wide  range  of  variables  are  included, 
this  approach  also  provides  results  having  greater  general izability  than 
single-factor  experiments. 

To  establish  possible  differences  among  the  above  factors,  the 
following  dependent  variables  were  studied:  time  until  return  of  the 

instrument,  percentage  of  instruments  returned,  average  ratings  for  job 
task  items,  and  finally,  a measure  of  questionnaire  completion  accuracy 
based  on  deviations  of  ratings  of  task  frequency  from  known  frequencies 
for  certain  job  tasks. 

SELECTION  OF  A SCHOOL/JOB  SETTING  FOR  THE  STUDY 

It  was  essential  to  identify  a training  program  which  actually 
exhibited  a substantial  number  of  each  of  the  three  classes  of  training 
problems  (undertraining,  overtraining,  and  irrelevant  training),  in 
order  to  establish  the  effectiveness  of  any  feedback  method,  and  to  make 
valid  comparisons  between  different  methods  and  different  groups  of 
respondents.  An  additional  requirement  for  the  program  was  that  a large 
number  of  school  graduates  be  available  to  obtain  reliable  comparisons 
among  a number  of  instruments,  groups  and  conditions. 

The  Radioman  "A"  School  most  nearly  met  the  criteria  stated  in  the 
paragraph  above  and  was  selected  as  the  test  bed  for  the  development  of 
"prototype"  feedback  instruments  and  procedures.  This  school  was  deemed 
most  suitable  since  at  the  time  of  initial  contact,  a major  curriculum 
revision  had  just  been  completed  based  on  a task  analysis  made  two  years 
previously.  The  new  curriculum  included  a number  of  important  new  job 
tasks  that  were  not  included  in  the  old,  and  thereby  corrected  many 
instances  of  undertraininq  which  existed  in  the  old  curriculum.  However, 
the  major  difference  between  the  old  and  new  curricula  was  that  separate 
tracks  were  instituted  for  shipboard  Radiomen  and  for  shore-based  Radiomen. 
The  separate  tracking  eliminated  a great  deal  of  overtraining  and 
irrelevant  training  on  shipboard  communications  that  existed  for  shore- 
based  personnel . This  also  eliminated  the  waste  of  training  resources 
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that  existed  for  shipboard  personnel  who  were  being  taught  a great  deal 
about  shore  communications  that  they  might  never  use  or  could  obtain 
later  in  their  careers  as  the  need  arose. 

Although  the  new  Radioman  "A"  School  curriculum  had  been  developed, 
it  had  not  yet  been  implemented  and  the  existing  curriculum  was  still 
producing  approximately  50  graduates  per  week  from  the  Bainbridge  and 
San  Diego  schools.  This  large  number  of  graduates,  plus  the  documented 
examples  of  training  problems,  made  this  an  ideal  setting  for  an  empirical 
comparison  of  feedback  methods.  In  addition,  school  personnel  and  the 
Training  Program  Coordinator  for  the  school  expressed  a willingness  to 
cooperate  and  aid  in  this  effort. 

IDENTIFICATION  AND  LOCATION  OF  SCHOOL  GRADUATES  AND  SUPERVISORS 

Prior  to  administering  feedback  instruments,  it  was  essential  to 
have  accurate  data  on  the  location  of  Radioman  "A"  School  graduates  and 
to  have  the  names  of  their  supervisors.  One  reason  for  this  was  that 
only  one  mail-out  instrument  would  be  mailed  to  a supervisor  regardless 
of  the  number  of  graduates  he  supervised.  It  was  feared  that  mailing 
more  than  one  instrument  to  a supervisor  would  be  burdensome  and  cause 
supervisor  resistance  to  the  study  which  in  turn  would  reduce  the  return 
rate  of  the  instruments.  Also,  accurate  pinpointing  of  graduate  locations 
would  insure  that  return  rate  statistics  would  be  less  influenced  by 
delayed  or  lost  questionnaires.  Finally,  one  of  the  key  variables  of 
the  study  was  the  ship  or  shore  location  of  the  graduate,  and  accurate 
identification  of  his  whereabouts  was  essential  for  correct  data  analysis 
on  this  ship/shore  duty  station  factor. 

To  obtain  these  data,  letters  were  prepared  and  mailed  to  the 
different  operational  units  receiving  Radioman  "A"  School  graduates. 
Communication  Officers  were  asked  to  confirm  whether  the  trainee  was 
stationed  at  the  unit  and  to  add  to  the  list  the  names  of  any  additional 
recent  graduates.  Finally,  they  were  asked  to  add  the  name  of  the 
supervisor  who  was  most  familiar  with  the  graduate's  performance  from 
the  date  of  his  arrival. 

Permission  to  contact  operational  units  was  obtained  from  the 
Atlantic  and  Pacific  Fleets  and  from  the  Naval  Telecommunications  Command. 
At  the  same  time,  permission  was  obtained  to  mail  the  feedback  instruments 
directly  to  the  trainee  and  to  the  supervisor.  Such  direct  contact 
reduced  handling  of  materials  by  command  personnel.  With  the  large 
number  of  persons  to  be  contacted  in  the  study,  such  handling  could  have 
become  burdensome  and  a potential  source  of  non-cooperation. 
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INSTRUMENTS 

Three  versions  of  the  mail-out  questionnaire  were  compared  in  this 
study.  These  were  (1)  a short  questionnaire  with  items  based  on  general 
job  tasks  including  several  open-ended  questions  to  get  at  specific 
training  problems,  (2)  a long  questionnaire  based  on  specific  job  tasks, 
including  one  open-ended  question,  and  (3)  a card  sort  procedure  using 
the  same  task  list  and  open-ended  question  as  were  used  for  the  long 
questionnaire.  Two  different  versions  were  prepared  for  each  of  these 
instruments,  one  for  the  trainees  and  one  for  the  supervisors.  In 
addition,  versions  of  each  were  prepared  with  normal  and  reversed  item 
order  as  a means  of  exploring  changes  in  the  accuracy  of  responding  as  a 
function  of  the  length  of  the  instrument.  Fatigue  or  boredom  effects 
would  show  up  as  differences  in  average  ratings  for  the  same  item  when 
it  was  encountered  early  in  one  version  and  later  in  another. 

All  three  mail -out  instruments  were  designed  to  provide  information 
about  undertraining,  overtraining,  training  relevancy,  and  missing 
training.  The  long  questionnaire  and  card  sort  instrument  provided 
information  on  the  first  three  training  problems  by  obtaining  ratings  of 
specific  job  tasks  that  were  listed  in  the  instrument.  Missing  training 
objectives  were  to  be  identified  by  the  open-ended  question.  Specific 
tasks  were  not  listed  on  the  short  questionnaire,  only  general  ones. 

Seven  open-ended  questions  were  added  to  get  at  undertraining,  over- 
training, and  training  relevancy  for  specific  job  tasks  as  well  as  to 
get  at  missing  training  objectives. 

SHORT  QUESTIONNAIRE.  The  15  general  task  statements  used  in  the  short 
questionnaire  are  presented  in  appendix  A.  At  a very  general  level 
these  describe  the  work  performed  by  Radiomen  at  Fleet  and  shore  units. 
These  items  were  typed  on  two  pages  of  the  form  presented  in  appendix  C. 
This  basic  form  (with  minor  changes)  was  used  on  both  the  long  and  short 
questionnaires  and  provided  the  three  scales  of  Frequency  of  Task, 
Criticality  of  Task,  and  Performance  of  Task  Upon  Arrival  From  School; 
space  for  eight  task  descriptions;  and  a brief  set  of  instructions.  The 
respondent  indicated  his  rating  on  these  scales  by  circling  a number 
corresponding  to  one  of  the  five  available  alternatives  for  each.  Seven 
open-ended  questions  (appendix  D)  were  added  at  the  end  of  the  short 
Questionnaire.  The  initial  page  of  the  questionnaire  was  a cover  letter 
(appendix  E).  It  was  followed  by  instructions  (appendix  F),  a biographical 
data  sheet  (appendix  G),  then  the  general  task  pages  and,  finally,  the 
open-ended  questions.  Appendixes  D through  G are  from  the  trainee 
version  of  the  short  questionnaire.  The  supervisor  version  of  these 
forms  differed  only  in  minor  changes  of  wording.  Time  to  complete  the 
short  questionnaire  was  about  15  minutes. 
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LONG  QUESTIONNAIRE.  The  long  questionnaire  utilized  the  134  specific 
job  tasks  presented  in  appendix  B.  Most  of  these  were  taken  from  both 
the  old  and  new  curricula.  Those  related  to  radiotelegraph  operation 
came  from  the  job  task  analysis,  but  are  not  included  in  either  the  old 
or  new  curricula.  The  questions  were  printed  on  20  pages  of  the  same 
basic  rating  scale  form  (appendix  C)  used  in  the  short  questionnaire, 
and  made  up  the  bulk  of  the  document.  The  same  cover  letter,  instructions, 
and  biographical  data  sheet  as  those  used  on  the  short  questionnaire 
preceded  these  tasks  and  rating  scales.  The  final  page  of  the  long 
questionnaire  was  a single  open-ended  question  (appendix  H)  designed  to 
identify  new  areas  of  instruction  which  should  be  added  to  the  curriculum 
of  Radioman  "A"  School.  Supervisor  versions  of  the  long  questionnaire 
presented  only  minor  modifications  of  the  forms  in  the  appendixes.  The 
time  to  complete  the  long  questionnaire  averaged  approximately  one  hour. 

CARD  SORT  PROCEDURE.  The  card  sort  procedure  used  the  same  134  items  as 
the  long  questionnaire.  These  were  printed  on  small  cards  (2.25  in.  x 4 
in.)  with  one  task  statement  per  card.  A "placemat"  contained  six 
rectangles  labeled  with  the  categories  into  which  the  134  cards  were  to 
be  sorted  and  also  included  the  instructions  for  this  sorting.  The 
"placemat"  is  presented  in  appendix  I.  Labeled  rubber  bands  were  enclosed 
to  be  used  following  sorting  of  the  cards  to  make  six  separate  packets 
that  were  to  be  returned  along  with  the  same  biographical  data  sheet 
(appendix  G)  and  open-ended  question  (appendix  H)  that  were  used  with 
the  long  questionnaire.  A Supervisor  "placemat"  with  minor  changes  was 
also  used.  Time  to  complete  this  card  sort  procedure  averaged  about  30 
minutes. 

STRUCTURED  INTERVIEW.  In  addition  to  the  mail -out  feedback  procedures, 
face-to-face  structured  interviews  were  conducted.  A long  questionnaire 
was  used  for  this  purpose  with  separate  copies  for  the  interviewer  and 
interviewee.  The  person  being  interviewed  was  asked  to  read  the  task 
statement  aloud  then  tell  his  ratings  on  the  three  scales  to  the  inter- 
viewer who  marked  these  on  his  copy  of  the  long  questionnaire.  After 
all  tasks  had  been  considered,  those  which  received  "inadequate"  Perform- 
ance of  Task  Upon  Arrival  From  School  ratings  were  reviewed  and  specific 
questions  about  training  for  these  tasks  asked.  Answers  were  recorded 
by  the  interviewer.  Suggestions  for  additional  material  to  be  included 
in  the  curriculum  were  also  requested  and  recorded.  This  procedure 
guaranteed  that  consideration  would  be  given  to  each  item  and  produced  a 
large  amount  of  discussion  relevant  to  training  for  the  different  tasks. 
Average  time  for  these  interviews  was  one  hour  and  twenty  minutes. 

SUBJECTS 

Instruments  were  mailed  to  996  trainees  and  590  supervisors  and 
structured  interviews  were  administered  to  an  additional  59  trainees  and 
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37  supervisors.  The  names  of  these  personnel  were  obtained  from  verified 
lists  of  graduates  returned  from  operational  units.  The  trainees  were 
graduates  of  the  unmodified  Radioman  "A"  School  curriculum  during  the 
period  May  1973  to  July  1974.  Almost  without  exception,  these  graduates 
were  early  in  their  first  enlistment  with  most  going  directly  from 
recruit  training  to  "A"  School.  Supervisors  were  usually  First  Class  or 
Chief  Petty  Officers  with  10  or  more  years  of  service.  The  different 
numbers  of  trainees  and  supervisors  resulted  from  the  fact  that  a super- 
visor often  supervised  more  than  one  trainee. 

More  of  these  personnel  were  stationed  at  shore  installations  (885) 
than  aboard  ships  (701)  and  the  difference  reflects  the  fact  that  a 
majority  of  Radioman  "A"  School  graduates  receive  their  first  assignment 
at  shore  stations.  Two  or  more  trainees  are  more  apt  to  have  the  same 
supervisor  if  the  trainee  is  at  a shore  installation  than  if  he  is 
aboard  ship.  This  difference  can  be  seen  in  table  1 which  presents  a 
breakdown  of  subjects  in  the  study  by  factors  of  supervisor  vs.  trainee, 
feedback  method,  and  duty  station. 


TABLE  1.  NUMBER  OF  TRAINEES  AND  SUPERVISORS  AT  DIFFERENT 
DUTY  STATIONS  RECEIVING  DIFFERENT  INSTRUMENTS 


Long 

Questionnaire 

Short 

Questionnaire 

Card 

Sort 

Structured 

Interview 

Trainees : 

Ship 

136 

144 

135 

18 

Shore 

192 

196 

193 

41 

Supervisors: 

Ship 

96 

101 

89 

19 

Shore 

100 

105 

99 

18 

25 


TAEG  Report  No.  19 


The  fact  that  many  trainees  shared  a supervisor  and  only  one  instru- 
ment was  mailed  to  a supervisor  resulted  in  another  imbalance  in  the 
numbers  of  subjects  in  the  joint  vs.  single  administration  conditions. 

Three  hundred  and  fourteen  supervisors  were  selected  randomly  and  assigned 
to  the  joint  administration  condition.  This  made  a total  of  622  trainees 
and  supervisors  receiving  joint  administration.  The  remaining  279 
supervisors  of  recent  graduates  received  instruments  but  recent  graduates 
they  supervised  did  not.  The  remaining  685  available  trainees  also 
received  instruments  but  their  supervisors  did  not  (unless  the  supervisor 
received  a form  to  complete  on  another  trainee).  This  made  a total  of 
622  persons  in  the  joint-administration  condition  compared  to  the  964 
with  single  administration. 

Approximately  equal  numbers  of  subjects  were  assigned  to  different 
levels  of  the  factors  of  Instrument  Type,  Class  Standing,  and  Item 
Order.  Differences  did  exist  in  the  number  of  subjects  in  the  different 
Time  Lapse  from  Graduation  groups.  These  differences  reflected  normal 
variations  in  graduate  output  during  the  year  and  also  the  inclusion  of 
graduates  from  a five-month  period  in  the  greater-than-1 2-months  time- 
lapse  group.  The  only  other  imbalance  that  appeared  in  numbers  for 
different  levels  of  a factor  was  in  the  Expectation  Letter  vs.  No  Expecta- 
tion Letter  groups.  This  resulted  because  not  all  data  requests  from 
operational  units  had  returned  before  questionnaire  mailing  began.  Late 
arriving  names  of  trainees  and  their  supervisors  were  mailed  question- 
naires, but  to  prevent  further  delays  in  their  mailing,  none  of  these 
personnel  received  the  prior  expectation  letter.  For  this  reason,  the 
number  of  questionnaires  preceded  by  "expectation  letters"  (758)  was 
smaller  than  the  number  mailed  without  these  letters  (828). 

The  imbalances  in  numbers  of  subjects  that  existed  in  many  conditions 
were  not  large  enough  to  reduce  the  reliability  of  data  from  smaller 
groups.  However,  data  from  these  groups  required  careful  analysis  and 
interpretation  to  avoid  confounding  of  effects  associated  with  one 
variable  with  the  effects  associated  with  another. 

PROCEDURE 

Four  to  six  weeks  prior  to  mailing  of  the  instruments,  approximately 
one-half  of  the  trainees  and  supervisors  were  mailed  a letter  announcing 
the  arrival  of  the  feedback  materials  and  requesting  consideration  of 
training  problems  in  Radioman  "A"  School.  In  the  joint  administration 
condition,  where  both  trainee  and  supervisor  were  to  be  mailed  instruments, 
both  persons  received  the  "expectation  letter"  or  neither  did.  The  bulk 
of  the  feedback  instruments  was  mailed  November  17  through  November  19, 

1974  with  the  remainder  being  mailed  over  the  next  month  as  additional 
verified  data  on  trainee  location  and  supervisors  were  returned  from 
operational  units.  Each  instrument  was  accompanied  by  a self-addressed 
return  envelope. 
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Follow-up  procedures  were  initiated  at  seven  and  at  nine  weeks 
after  mailing  of  the  evaluation  materials.  The  first  follow-up  to  non- 
respondents was  a simple  reminder  letter  requesting  the  return  of  the 
questionnaire.  The  second  requested  that  they  indicate  their  reasons 
for  not  returning  the  materials  and  asked  for  suggestions  for  improve- 
ment in  future  evaluation  efforts.  A follow-up  letter  was  also  mailed 
to  those  persons  who  had  completed  and  returned  the  questionnaire  asking 
them  to  relate  any  problems  they  may  have  had  completing  the  materials 
and  to  suggest  improvements  for  future  feedback  efforts. 

As  instruments  were  returned,  the  rating  scale  data  were  immediately 
punched  on  cards  for  analysis.  A seven-digit  identification  code  was 
included  on  these  cards  that  designated  the  status  of  the  respondent  on 
the  variables  of  Trainee  vs.  Supervisor,  Joint  vs.  Single  Administration, 
Questionnaire  type,  and  the  remaining  five  variables  studied.  When  it 
appeared  that  no  more  instruments  were  forthcoming,  statistics  were 
calculated  including  return  rates  for  the  various  conditions  and  sub- 
conditions, averages  and  standard  deviations  of  rating  scale  data,  and 
similar  statistics  on  time  to  return  and  numbers  of  comments. 
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SECTION  III 
RESULTS 

Results  are  presented  for  the  major  dependent  variables  of  instrument 
return  rate,  responses  on  the  rating  scales,  responses  to  open-ended 
questions,  and  the  results  of  follow-up  efforts  that  took  place  after 
the  main  comparison  of  feedback  methods.  Descriptive  statistics  are 
tabulated  and  the  statistically  significant  differences  are  described  in 
the  text.  The  decision  rule  selected  for  statistical  significance  was 
that  a result  should  occur  by  chance  only  five  times  or  less  out  of  100 
(p  <.05) . 

INSTRUMENT  RETURN  STATISTICS 

The  instrument  return  statistics  include  the  overall  return  rate 
for  instruments,  the  time  from  mailing  until  questionnaire  return,  and 
the  percentage  of  persons  receiving  reminder  letters  who  responded 
following  the  reminder  letter.  These  instrument  return  statistics  are 
described  below  for  each  of  the  factors  included  in  the  study.  The 
order  of  presentation  of  factors  is  generally  related  to  the  magnitude 
and  importance  of  the  observed  differences. 

RESPONDENTS.  Statistics  on  instrument  return  for  trainees  and  supervisors 
are  presented  in  the  top  part  of  table  2.  The  return  rate  for  supervisors 
was  almost  double  that  of  trainees  and  the  difference  was  highly  significant 
(z  = 5.2,  p<.001).  The  percentage  of  reminder  letters  producing  returned 
instruments  was  also  much  higher  for  supervisors  than  trainees  and 
highly  significant  (z  = 6.5,  p<.001).  The  difference  in  average  time 
from  mailing  to  return  was  not  significant  for  these  two  groups. 

INSTRUMENTS.  The  data  for  instruments  are  presented  in  the  middle  of 
table  2.  There  was  a significantly  higher  return  rate  for  the  short 
questionnaire  than  for  either  the  long  questionnaire  or  card  sort  proce- 
dure (z  = 3.7  for  short  vs.  long;  z = 3.4  for  short  vs.  card  sort;  p<.001 
for  both).  The  small  difference  between  the  long  questionnaire  and  the 
card  sort  instrument  was  not  significant.  Differences  among  the  three 
instruments  for  the  percentage  of  reminder  letters  producing  returned 
questionnaires  also  were  not  significant.  However,  the  longer  average 
time  to  return  for  the  long  form  than  the  short  form  was  significant 
(t  = 3.2,  df  = 460,  p<.01),  as  was  the  difference  in  time  to  return 
between  the  long  form  and  the  card  sort  instrument  (t  = 2.0,  df  = 392, 

P < • 05 ) . 

Instrument  return  data  on  the  combined  factors  of  Instrument  Type 
and  Respondent  are  presented  in  the  bottom  part  of  table  2.  For  trainees, 
the  return  rate  for  the  short  form  was  significantly  higher  than  for  the 
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TABLE  2.  NUMBER  OF  INSTRUMENTS  MAILED  AND  RETURNED  AND  AVERAGE 
TIME  TO  RETURN  FOR  DIFFERENT  RESPONDENT  GROUPS  AND 
INSTRUMENTS 


/ 


/ 


/ 


Instrument 

or 

Group 


Qj  *</ 

'i  £*  fe  S S , i ££  M H $ 


0; 

/j7 

i- r £/&££/j?cr 


Qj  ^ O 

£ o/<=>  ^ 


Trainee 

Respondent 

Supervisor 

996 

590 

310 

350 

31.1 

59.3 

43 

66 

j 

5.9 

21.6 

| 

32.1 

35.1 

Short  Form 

546 

266 

48.7 

38 

11.9 

30.7 

■ Instrument  Long  Form 

524 

196 

37.4 

43 

11.6 

38.8 

Card  Sort 

516 

198 

38.4 

28 

8.1 

33.2 

Short-Trainee 

1 

340 

1 

136 

40.0 

19 

8.5 

31.0 

Long-Trainee 

328 

83 

25.2 

14 

5.4 

36.2 

Respondent  Card  Sort-Trainee 

> 

328 

91 

27.7 

10 

4.0 

32.2 

1 

X Short- Supervisor 

206 

130 

63.1 

19 

20.0 

30.4 

Instrument  Long-Supervisor 

196 

113 

57.7 

29 

25.9 

41.3 

Card  Sort- 

Supervisor 

188 

107 

i 

. 

56.9 

18 

1 

J 

18.2 

34.0 

l 
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long  form  or  the  card  sort  ( z = 4.0  for  short  vs.  long;  z = 3.3  for 
short  vs.  card  sort;  p<.001  for  both).  In  addition,  for  trainees  the 
percentage  of  short  questionnaires  returned  as  a result  of  reminder 
letters  was  significantly  higher  than  the  corresponding  percentage  for 
the  card  sort  instrument  (z  = 2.0,  p<.05).  For  supervisors,  the  corre- 
sponding differences  between  instruments  were  considerably  smaller  and 
not  statistically  significant. 

TIME  LAPSE  PRIOR  TO  EVALUATION.  Data  for  groups  with  different  times 
from  graduation  prior  to  evaluation  are  presented  in  the  top  half  of 
table  3.  The  4 to  6 months  group  yielded  a lower  return  rate  than  each 
of  the  two  longest  periods  (z  = 2.6  for  the  4 to  6 vs.  10  to  12  groups; 
z = 2.5  for  the  4-6  vs.  greater  than  12  groups;  p^.05).  The  other 
return  rate  differences  among  these  groups  were  not  significant.  None 
of  the  differences  in  time  to  return  and  percentage  of  instruments 
returned  following  the  reminder  letters  were  significant  for  these 
groups. 

ADMINISTRATION  PROCEDURE.  Instrument  return  statistics  are  presented  at 
the  bottom  of  table  3 for  the  factor  of  Joint  vs.  Single  Administration 
of  the  instruments.  Unequal  numbers  of  trainees  and  supervisors  in  the 
single  administration  condition  required  the  breakdown  of  these  results 
by  the  Respondent  Group  factor.  Although  a higher  return  rate  in  favor 
of  joint  administration  appeared  for  both  trainees  and  supervisors,  it 
was  not  significant  (z  = 1.6,  p < . 1 1 ) . No  significant  differences 
occurred  between  joint  and  single  administration  on  the  other  return 
statistics. 

CLASS  STANDING.  Instrument  return  statistics  are  presented  in  table  4 
for  the  different  Class  Standing  groups  with  separate  data  presented  for 
trainees  and  supervisors.  The  significant  differences  for  trainees  were 
on  the  percent  of  instruments  returned  and  on  the  percent  of  successful 
reminder  letters.  Trainees  in  the  bottom  third  of  their  graduating 
classes  returned  significantly  fewer  instruments  than  trainees  in  the 
upper  and  middle  third  of  their  classes  (z  = 3.3  for  lower  1/3  vs.  upper 
1/3;  z = 2.7  for  lower  1/3  vs.  middle  1/3;  p ^.01  for  both).  Trainees 
in  the  lower  third  of  their  classes  also  accounted  for  a lower  percentage 
of  reminder  letters  producing  return  than  was  found  for  trainees  in  the 
upper  third  of  their  classes  (z  = 2.3,  p<.05).  Supervisors  took  signifi- 
cantly longer  to  return  instruments  for  the  lower  third  of  graduates 
than  the  other  groups  (t  = 2.74,  df  = 238  p < .01  for  lower  1/3  vs.  upper 
1/3;  t = 2.23,  df  = 221,  p<.05  for  lower  1/3  vs.  middle  1/3). 
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TABLE  3.  NUMBER  OF  INSTRUMENTS  MAILED  AND  RETURNED  AND  AVERAGE 

TIME  TO  RETURN  FOR  DIFFERENT  TIME  FROM  GRADUATION  GROUPS 
AND  ADMINISTRATION  PROCEDURES 


Experimental 

Condition 

or 

Group 


Time 

4-6 

340 

t 

120 

35.3 

l 

, 19 

7.9 

’ 

j 34.1 

Lapse 

Prior 

7-9 

408 

168 

41.2 

32 

1 

111. 8 

35.8 

To 

Evaluation 

10-12 

279 

127 

45.5 

19 

in.i 

33.4  ' 

(Months) 

More  than  12 

559 

' 

’ 245 

i 

43.8 

| 39 

11.0 

1 

34.7  j 

Admi ni stra- 

Joint - Trainee 

311 

' 106 

34.1 

16 

7.2 

] 31.4 

ti  ve 

Procedure 

Single  - Trainee 

685 

no 

o 

-c* 

29.8 

27 

5.3 

1 

' 32.6 

• 

Joint-Supervisor 

311 

! 192 

61.7 

38 

24.2 

! 35.5 

i 

Single-Supervisor 

279 

! 158  ; 

56.6 

28 

18.8 

j 33.7  i 
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TABLE  4.  NUMBER  OF  INSTRUMENTS  MAILED  AND  RETURNED  AND  AVERAGE 
TIME  TO  RETURN  FOR  DIFFERENT  CLASS  STANDING  GROUPS 


Trainee 

Class 

Standing 

Upper  1/3 
Middle  1/3 
Lower  1/3 

320 

333 

343 

115 

112 

83 

35.9 

33.6 

24.2 

18 

16 

9 

8.1 

6.8 

3.3 

35.8 

31.9 
28.1 

Supervisor 

Class 

Upper  1/3 

212 

127 

59.9 

22 

20.6 

28.2 

Standing 

I 

Middle  1/3 

176  | 

no 

62.5 

14 

17.5 

30.5 

J 

i 

i 

Lower  1/3 

J 

l 

202 

i 

i 

113 

55.9 

31 

l 

25.8 
1 

40.5  1 

l 

i 
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OTHER  FACTORS.  Instrument  return  statistics  for  the  factors  of  Duty 
Station,  Item  Order,  and  Expectation  Group  are  presented  in  table  5. 

None  of  the  differences  between  ship  and  shore  duty  station  were  signifi- 
cant for  any  of  the  dependent  variables.  This  was  also  true  for  instruments 
with  normal  item  order  as  opposed  to  reversed  item  order.  Significantly 
more  instruments  were  returned,  however,  for  the  group  receiving  "expecta- 
tion letters"  before  the  questionnaires  than  for  the  group  that  did  not 
receive  these  letters  (z  = 2.0,  p<.05).  When  data  from  trainees  and 
supervisors  were  considered  separately  for  the  "expectation  letter" 
factor,  only  the  return  rate  difference  for  trainees  was  significant  (z 
= 1 .8,  p<.05). 

RATING  SCALE  DATA 

Since  the  rating  scale  data  are  the  prime  means  for  identification 
of  training  successes  and  failures  for  different  tasks,  they  are  also 
the  key  data  for  judging  instrument  effectiveness  in  the  comparison  of 
instruments  and  respondent  groups.  For  all  rating  scales  the  scale 
values  ranged  from  one  to  five.  High  frequency  of  tasks,  high  criticality 
of  tasks,  and  good  performance  of  tasks  were  indicated  by  high  numbers 
on  the  respective  scales. 

DIFFERENCES  AMONG  ITEMS.  The  largest  source  of  variance  in  all  comparisons 
was  among  the  items  themselves.  This  was  particularly  true  on  the 
Frequency  of  Task  scale  where  overall  means  ranged  from  1.25  (item  No. 

124  of  appendix  B)  to  4.6  (item  Mo.  6 of  appendix  A).  Since  only  a 
difference  of  four  units  existed  on  any  of  the  five-point  scales,  this 
range  of  3.35  units  for  ratings,  averaged  over  all  subjects,  indicated 
an  extremely  accurate  and  consistent  pattern  of  responding  by  nearly  all 
the  personnel  who  returned  questionnaires.  Even  more  extreme  average 
ratings  occurred  when  ship  and  shore  results  were  considered  separately. 
Nearly  one-third  of  the  items  showed  average  Frequency  ratings  above  4.5 
or  below  1.5. 

The  accuracy  and  consistency  of  ratings  were  also  dramatically 
illustrated  in  the  high  correlations  found  when  average  item  ratings 
were  correlated  between  different  instruments,  between  respondent  groups, 
and  even  between  the  ratings  on  the  different  scales  (e.g..  Frequency 
with  Criticality).  Correlations  between  average  item  ratings  for  the 
three  rating  scales  are  presented  in  table  6.  It  can  be  seen  that  these 
correlations  between  scales  on  the  short  form  were  all  nearly  1.00. 

Although  these  correlations  were  also  high  on  the  long  form  of  the 
questionnaire,  each  was  significantly  lower  than  the  corresponding 
correlation  for  the  short  form  (p<:05).  The  lower  correlations  for  the 
long  questionnaire  indicate  a more  accurate  description  by  this  instrument 
of  the  known  differences  that  exist  among  the  variables  of  task  frequency, 
task  criticality,  and  trainee  task  performance. 
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TABLE  5.  NUMBER  OF  INSTRUMENTS  MAILED  AND  RETURNED  AND  AVERAGE 
TIME  TO  RETURN  FOR  DIFFERENT  DUTY  STATIONS,  ITEM  ORDER, 
AND  EXPECTATION  GROUPS 


Duty 

Station 

• 

Shore 

Ship 

1 

885 

701 

' 

373 

287 

! 

42.1 

40.9 

53 

56 

9.4 

11.9 

\ ' 

32.3 

36.2 

Item 

Normal  Order 

803 

348 

43.3 

62 

1 

12.0 

36.2 

Order 

Reversed  Order 

783 

312 

39.8 

47 

9.1 

32.0 

. 

Expectation 

Letters 

Letter,  Trainee 
& Supervisor 

756 

335 

44.2 

57 

11.9 

33.7 

No  Letter,  Trainee 
& Supervisor 

i 

828 

1 

1 

325 

39.3 

52 

9.4 

32.7 

Letter,  Trainee 

471 

160 

33.9 

19 

5.8 

30.0 

No  Letter,  Trainee 

525 

150 

28.6 

24 

6.0 

'34.4  | 

Letter,  Supervisor 

287 

175 

61.0 

38  i 

! 25.3 

37.0 

No  Letter, 
Supervisor 

| 

303 

! 

175 

1 

57.8 

1 1 

; 28 

1 

17.9 

1 I 

31.3 
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TABLE  6.  CORRELATIONS  BETWEEN  AVERAGE  RATINGS  FOR  DIFFERENT 
RATING  SCALES  ON  LONG  AND  SHORT  QUESTIONNAIRES 


Instrument 

Frequency  With 

Frequency  With 

Criticality  With 

1 

Criticality 

Performance 

Performance 

1 

* 

< 

i Short 

Questionnaire 

.93 

.97 

i 

1 

i 

.94 

i , 

i Long 

j Questionnaire 

1 

.70 

.84 

.70 

The  patterns  of  average  ratings  for  items  on  the  Performance  of 
Task  Upon  Arrival  From  School  scale  for  the  long  questionnaire  and  card 
sort  were  nearly  identical,  the  correlation  between  the  two  instruments 
for  the  134  items  being  .89.  The  magnitude  of  this  correlation  implies 
that  the  differences  between  average  ratings  on  performance  of  different 
items  are  real  differences  and  not  chance  occurrences. 

Performance  ratings  for  several  of  the  items  that  were  not  present  in 
the  old  Radioman  "A"  School  curriculum  were  significantly  lower  than  the 
average  Performance  rating  for  all  items.  For  example,  item  4 on  the 
long  questionnaire,  which  dealt  with  reading  perforated  teletype  tapes 
had  an  average  rating  of  2.15  which  was  significantly  lower  than  the  3.31 
overall  average  rating  of  Performance  ( t = 7.9,  df  = 390,  p<.001).  In 
addition,  the  nine  items  that  dealt  with  the  use  of  International  Morse 
Code  (items  122  through  130)  produced  an  average  Performance  rating  of 
2.45,  which  was  significantly  lower  than  the  overall  Performance  average 
(t  = 7.7,  df  = 390,  p<.001). 

SHIP/SHORE  DUTY  STATION  DIFFERENCES.  The  second  largest  source  of  variance 
in  the  rating  scale  data  was  produced  by  the  duty  station  of  the  trainee 
(see  table  7).  On  both  the  Frequency  of  Task  and  Criticality  of  Task 
scales  significantly  higher  ratings  were  found  for  ship  personnel  than 
shore  personnel  (t  = 11.0,  df  = 658,  p^.001  for  Frequency;  t = 8.6, 
df  = 658,  p-sC.001  for  Criticality).  However,  average  ratings  on  the 
Performance  of  Task  Upon  Arrival  From  School  scale  were  nearly  identical 
for  shore  based  and  shipboard  personnel. 
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TABLE  7.  AVERAGE  SCALE  RATINGS  OF  SHIP  AND  SHORE  DUTY 
STATION  RESPONDENTS,  TRAINEES,  AND  SUPERVISORS 


Respondent 

Frequency  of 
Task  Scale 

Criticality  of 
Task  Scale 

Performance  of 
Task  upon  Arrival 
from  School  Scale 

Shore  Duty 
Station 

2.64 

3.33 

3.18 

Ship  Duty 
Station 

3.31 

3.88 

3.19 

Trainee 

3.02 

3.62 

3.34 

Supervisor 

2.95 

3.58 

3.05 

Correlations  between  shipboard  and  shore-based  personnel  for  the 
average  item  ratings  on  the  different  scales  of  the  long  questionnaire 
were  .72  for  Frequency,  .70  for  Criticality,  and  .88  for  Performance  of 
Task  Upon  Arrival  From  School.  The  Performance  correlation  was  significantly 
higher  (p<.05)  than  the  Frequency  and  Criticality  correlations.  The 
high  Performance  correlation  reflects  a high  degree  of  similarity  of 
Performance  ratings  for  the  two  groups,  which  would  be  expected  since 
both  received  the  same  training. 

The  large  ship/shore  differences  in  ratings  of  Frequency  for  many 
items  correspond  to  the  known  differences  in  the  job  structure  for 
Radiomen  between  these  different  sites.  The  accuracy  of  these  Frequency 
ratings  was  used  to  evaluate  different  groups  of  respondents  on  their 
instrument  completion  performance  and  these  results  are  discussed  later 
in  this  section. 

TRAINEE-SUPERVISOR  DIFFERENCES.  As  shown  in  table  7,  the  overall  average 
ratings  on  the  Frequency  and  Criticality  scales  were  nearly  identical 
for  trainees  and  supervisors.  However,  trainees  provided  significantly 
higher  average  ratings  on  the  Performance  of  Task  Upon  Arrival  From 
School  scale  than  supervisors  (t  = 4.9,  df  = 658,  p <T- 001 ) . Despite 
this  difference,  the  basic  pattern  of  item  ratings  was  highly  similar 
for  the  two  groups.  This  similarity  was  shown  in  the  correlations 
between  trainees  and  supervisors  on  the  average  Performance  ratings  for 
items  which  were  greater  than  .80  for  both  the  long  form  and  card  sort 
and  greater  than  .90  for  the  short  questionnaire. 
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Some  tasks  included  on  the  questionnaire  are  performed  very  often 
by  Radiomen  and  some  are  seldom  performed.  These  tasks  were  a source  of 
rating  differences  between  the  trainees  and  supervisors,  with  supervisors 
tending  to  give  the  more  extreme  Frequency  ratings  on  such  tasks.  Such 
items  are  included  in  the  Frequency-Accuracy  Index  discussed  below.  The 
more  extreme  Frequency  ratings  for  these  particular  items  corresponded 
to  the  known  frequencies  for  these  tasks,  and  supervisors  were  more 
accurate  than  trainees  in  describing  these  known  frequencies. 

CLASS  STANDING.  Class  standing  ratings  are  shown  in  table  8.  Statisti- 
cally significant  differences  in  average  Performance  of  Task  Upon  Arrival 
From  School  appeared  among  the  different  groups.  Graduates  in  the  upper 
third  of  the  classes  and  their  supervisors  rated  Performance  higher  than 
graduates  and  supervisors  of  the  other  two  groups.  The  difference 
between  the  upper  third  and  middle  third  was  significant  (t  = 3.4,  df  = 
462,  p<.001),  as  was  the  difference  between  the  upper  third  and  lower 
third  (t  = 3.6,  df  = 436,  p<.001).  The  difference  between  the  middle 
third  and  lower  third  was  not  significant. 


TABLE  8.  AVERAGE  SCALE  RATINGS  FOR  CLASS  STANDING  GROUPS 


Class  Standing 

Frequency  of 
Task  Scale 

Criticality  of 
Task  Scale 

Performance  of 
Task  Upon  Arrival 
From  School  Scale 

Upper  third 

2.99 

3.65 

3.35 

Middle  third 

2.94 

3.57 

3.12 

Bottom  third 

3.02 

3.58 

3.09 

TIME  LAPSE  FROM  GRADUATION.  There  were  significant  differences  in 
Frequency  of  Task  ratings  dependent  upon  the  amount  of  time  lapse  from 
graduation  prior  to  interrogation  (table  9).  The  4 to  6 month  group 
provided  significantly  lower  average  Task  Frequency  ratings  than  the  7 to  9 
month  group  (t  = 3.0,  df  = 286,  p<.01).  In  turn,  the  7 to  9 month 
group  provided  significantly  lower  average  Task  Frequency  ratings  than 
the  greater  than  12  month  group  (t  = 1.7,  df  = 411,  p<T.05).  The  trend 
is  clearly  one  of  higher  ratings  of  Frequency  of  Task  with  more  time  lapse 
from  graduation.  Differences  between  these  groups  on  the  other  scales 
were  not  significant. 
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TABLE  9.  AVERAGE  SCALE  RATINGS  FOR  TIME  FROM  GRADUATION  GROUPS 


Time  Lapse 
Prior  to 
Evaluation 

Frequency  of 
Task  Scale 

Criticality  of 
Task  Scale 

Performance  of 
Task  Upon  Arrival 
From  School  Scale 

4 to  6 months 

2.71 

3.51 

3.27 

7 to  9 months 

2.99 

3.61 

3.08 

10  to  12  months 

3.02 

3.66 

3.18 

More  than  12 
months 

3.12 

3.66 

3.19 

INSTRUMENTS.  Rating  scale  data  for  the  different  instruments  are  presented 
in  table  10.  The  short  questionnaire  produced  significantly  higher 
average  Frequency  of  Task  ratings  than  the  long  questionnaire  (t  = 5.4, 
df  = 461,  p<£_.001).  On  the  Performance  of  Task  Upon  Arrival  From  School 
scale  the  average  rating  from  the  short  questionnaire  was  significantly 
lower  than  the  long  questionnaire  (t  = 5.6,  df  = 461,  p <. 001 ) . Both  of 
these  differences  probably  reflected  the  different  items  that  made  up 
the  two  different  instruments  and  were  not  related  to  the  length  or  any 
other  general  characteristics  of  these  instruments.  The  absence  of  a 
difference  in  Performance  ratings  for  the  long  form  and  card  sort  would 
be  expected  since  both  utilized  the  same  134  items. 


TABLE  10.  AVERAGE  SCALE  RATINGS  FOR  DIFFERENT  INSTRUMENTS 


Instrument 

Frequency  of 
Task  Scale 

Criticality  of 
Task  Scale 

Performance  of 
Task  Upon 
Arrival  From 
School  Scale 

Short  Form 

3.17 

3.68 

2.93 

Long  Form 

2.78 

3.52 

3.31 

Card  Sort 

N/A 

N/A 

3.32 

39 


TAEG  Report  No.  19 


ITEM  ORDER.  Average  Performance  on  the  normal  item  order  instruments 
was  lower  than  average  Performance  on  the  reversed  item  order  instruments. 
However,  this  difference  was  caused  primarily  by  the  significant  difference 
between  the  average  Performance  ratings  on  the  normal  order  (2.79)  and 
reversed  order  (3.09)  short  questionnaires  (t  * 2.9,  df  = 269,  p^.Ol). 

The  correlation  between  the  normal  and  reversed  forms  for  Performance 
ratings  of  the  15  short  questionnaire  items  was  .94,  which  indicates 
that  despite  the  difference  in  level,  relationships  between  items  were 
nearly  identical  for  the  two  forms.  The  normal  item  order  and  reversed 
item  order  versions  of  the  long  questionnaire  also  produced  nearly 
identical  data.  The  correlation  between  the  two  versions  was  .89  for 
Performance  ratings. 


TABLE  11.  AVERAGE  SCALE  RATINGS  FOR  ITEM  ORDER,  ADMINISTRATIVE 
PROCEDURE,  AND  EXPECTATION  LETTER  CONDITIONS 


Instrument 

Frequency  of 
Task  Scale 

Criticality  of 
Task  Scale 

Performance  of 
Task  Upon 
Arrival  From 
School  Scale 

Normal  Item 
Order 

2.95 

3.62 

3.12 

Reversed  Item 
Order 

3.01 

3.58 

3.27 

Single  Admin. 

2.99 

3.63 

3.21 

Joint  Admin. 

2.97 

3.57 

3.16 

Expectation 

Letter 

3.00 

3.59 

3.26 

No  Expectation 
Letter 

2.97 

3.61 

3.15 

ACCURACY  OF  FREQUENCY  RATINGS.  As  noted  earlier,  the  overall  average 
Frequency  rating  was  higher  for  shipboard  personnel  than  for  personnel 
at  shore  stations.  This  difference  was  produced  in  large  part  by  eight 
items  related  to  operation  of  the  Fleet  Broadcast  and  by  four  items 
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related  to  maintenance  performance  on  communications  equipment.  Other 
items  such  as  those  related  to  radiotelegraph  operation  had  been  revealed 
by  task  analysis  to  have  very  low  frequencies  for  both  ship  and  shore 
personnel.  Two  items  related  to  security  practices  were  also  known  to 
have  very  high  frequencies  for  both  ship  and  shore  personnel  as  did  a 
number  of  items  have  low  frequencies  of  occurrence  only  for  shore-based 
personnel.  From  all  these  tasks  with  previously  ascertained  very  high 
or  very  low  frequencies  of  occurrence  for  ship  personnel,  shore  personnel, 
or  both,  78  were  selected  to  provide  a measure  of  the  relative  accuracy 
of  Frequency  ratings  for  different  groups  and  conditions  of  the  study. 

On  each  item  comparisons  were  made  for  pairs  of  conditions,  such  as 
supervisors  vs.  trainees.  If  the  average  Frequency  rating  was  more 
extreme  (and  therefore  more  accurate)  for  the  trainee,  he  received  a one 
for  the  item  and  the  supervisor  received  a zero.  This  was  done  for  all 
78  items  and  a total  score  for  each  condition  was  obtained,  with  the 
condition  having  the  higher  number  being  more  accurate. 

When  average  ratings  for  trainees  on  these  78  tasks  were  compared 
to  the  average  ratings  for  supervisors,  only  21  of  the  78  items  were 
rated  more  accurately  by  the  trainees.  The  sign  test  (Siegel,  1956) 
indicated  that  significantly  more  items  were  rated  more  accurately  by 
supervisors.  (The  correlation  between  trainees  and  supervisors  for  the 
134  Frequency  ratings  was  .90  for  shipboard  personnel  and  .93  for  personnel 
at  shore  stations.  Thus,  despite  reduced  accuracy  of  responding,  trainees 
still  nearly  replicated  the  same  inter-item  relationships  produced  by 
supervisor  ratings. ) 

The  joint  vs.  single  administration  factor  produced  a significant 
difference  in  the  accuracy  of  Frequency  ratings.  Fifty-one  of  the  78 
tasks  were  rated  more  accurately  by  persons  receiving  the  long  question- 
naire jointly  with  the  supervisor  (or  trainee)  than  by  persons  in  the 
single  administration  condition  (p^.01,  sign  test). 

Another  significant  difference  in  Frequency  rating  accuracy  was 
found  for  the  lower  of  the  three  Class  Standing  groups.  Average  ratings 
for  this  group  were  significantly  less  accurate  than  the  overall  ratings 
on  53  of  these  78  items  (p<.01,  sign  test).  The  middle  third  and  upper 
third  groups  did  not  differ  between  themselves,  but  average  ratings  of 
both  were  more  accurate  than  overall  average  ratings  on  50  of  the  78 
items  (p<.01). 

Two  of  the  Time  from  Graduation  groups  produced  significant  differences 
in  accuracy  of  ratings  for  these  78  items.  The  group  who  received 
instruments  4 to  6 months  after  graduation  was  significantly  more  accurate 
than  the  overall  average  on  55  of  these  items  (p  < . 01 ) . The  group  that 
received  instruments  10  to  12  months  after  graduation  was  less  accurate 
than  the  overall  average  on  53  of  these  items  ( p < . 01 ) . No  other 
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differences  between  groups  and  conditions  in  the  study  appeared  for  this 
measure  of  the  accuracy  of  Frequency  ratings. 

STRUCTURED  INTERVIEW  SIMILARITIES  AND  DIFFERENCES.  The  patterns  of  item 
ratings  were  nearly  identical  between  the  interview  and  mail-out  data, 
expecially  in  the  case  of  ship  personnel.  The  correlation  between 
Frequency  ratings  from  the  two  procedures  was  .74  for  shore-based  person- 
nel and  .94  for  ship  personnel.  For  Criticality  ratings,  the  correlation 
was  .68  for  shore  personnel  and  .89  for  shipboard  personnel.  For  Perform- 
ance upon  Arrival  from  School,  the  correlations  were  .89  for  shore 
personnel  and  .92  for  shipboard  personnel.  The  lower  Frequency  and 
Criticality  correlations  for  shore  personnel  reflect  the  somewhat  atypical 
equipment  and  procedures  encountered  at  the  Norfolk  Communication  Station. 

As  with  rating  averages,  variability  of  ratings  was  also  nearly  identical 
for  persons  receiving  mail -out  instructions  and  for  persons  providing 
data  in  the  structured  interviews. 

One  difference  which  occurred  between  ratings  for  structured  inter- 
views and  mail -out  questionnaires  was  a greater  tendency  for  the  structured 
interview  group  to  skip  the  Criticality  and  Performance  scales  if  the 
task  was  "Never  Performed."  It  was  difficult  to  insist  that  a person 
make  a Performance  rating  on  a task  when  he  (or  the  person  he  supervised) 
had  never  performed  this  task  on  the  job.  Mail -out  questionnaires 
contained  instructions  to  do  this  and  respondents  were  generally  willing 
to  provide  such  ratings.  In  the  face-to-face  interview,  the  problem  of 
making  these  ratings  was  a shared  problem,  and  the  joint  decision  was 
often  to  not  make  such  ratings  when  there  was  little  or  no  experience 
upon  which  to  base  them. 

TRAINING  ADEQUACY  INDEX.  Tasks  with  high  ratings  of  Criticality  and  low 
ratings  of  Performance  Upon  Arrival  From  School  are  tasks  which  are  apt 
to  be  undertrained.  Conversely,  tasks  are  apt  to  be  overtrained  if  they 
receive  low  ratings  of  Criticality  and  high  ratings  of  Performance.  The 
134  tasks  of  the  long  questionnaire  were  examined  and  those  with  the 
largest  differences  between  average  Criticality  and  Performance  ratings 
were  identified.  Tasks  with  large  positive  differences,  i.e.,  greater 
Criticality  than  Performance,  were  classified  as  undertrained  on  this 
Training  Adequacy  Index.  Tasks  with  large  negative  differences;  i.e., 
greater  Performance  than  Criticality,  were  classified  as  overtrained  on 
this  index.  This  procedure  corresponds  closely  to  a technique  developed 
by  Siegel,  Schultz  and  Federman  (1961)  for  combining  Task  Criticality 
and  Task  Performance  ratings  to  obtain  a measure  of  training  adequacy. 

Since  training  problems  in  the  old  Radioman  "A"  School  curriculum 
centered  around  the  duty-station  factor,  the  procedure  was  carried  out 
separately  for  average  ratings  provided  by  trainees  and  supervisors  at 
shore  duty  stations  and  for  average  ratings  provided  by  shipboard  personnel. 
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This  procedure  indicated  four  tasks  (items  14,  36,  48  and  72  in  appendix 
B)  which  personnel  from  both  duty  stations  classified  as  overtrained, 
and  six  tasks  (items  74,  75,  102,  103,  123,  and  124)  which  personnel 
from  both  duty  stations  classified  as  undertrained.  Shipboard  personnel 
classified  an  additional  eight  tasks  as  overtrained  (items  15,  35,  50, 

51,  63,  67,  108  and  115)  and  an  additional  eight  tasks  as  undertrained 
(items  3,  20,  94,  101,  110,  122,  127  and  128).  Shore-based  personnel 
classified  an  additional  nine  tasks  as  overtrained  (items  40,  41,  43, 

45,  46,  47,  82,  83  and  84)  and  an  additional  seven  tasks  as  undertrained 
(items  4,  57,  112,  131,  132,  133  and  134).  Agreement  of  the  two  duty 
station  groups  on  only  10  of  the  42  items  augurs  well  for  the  derived 
Training  Adequacy  Index,  considering  the  large  differences  in  training 
needs  for  the  two  duty  station  groups  and  the  fact  that  the  old  curriculum 
was  identical  for  both. 

The  tasks  rated  as  overtrained  by  both  duty  station  groups  involved 
those  dealing  with  assignment  and  recording  of  date-time  groups  of 
messages,  delivering  messages,  and  changing  teletype  paper,  tapes  and 
ribbons.  Unlike  many  training  tasks  which  receive  little  attention  in 
school,  these  tasks  are  practiced  frequently  since  they  are  included  in 
performance  of  many  other  training  tasks  dealing  with  message  handling 
and  teletype  operation.  For  this  reason,  these  tasks  are  overlearned; 
i.e.,  more  practice  is  given  on  the  task  after  criterion  performance  has 
been  reached.  However,  this  overlearning  is  a by-product  of  other 
essential  training. 

Of  the  items  which  both  duty  station  groups  classified  as  under- 
trained, one  dealt  with  rescuing  a person  from  a live  circuit,  one  with 
first  aid,  two  with  distress  messages  and  enemy  contact  reports,  and  two 
with  restoring  fading  communication  links.  All  can  be  considered  emergency 
situations  and  at  first  glance  appear  to  be  likely  candidates  for  increased 
training  emphasis.  However,  no  particular  increased  emphasis  is  placed 
on  these  tasks  in  the  new  Radioman  "A"  School  curriculum  over  that  of 
the  old  curriculum. 

For  personnel  at  shore  duty  stations  the  derived  Training  Adequacy 
Index  indicated  overtraining  for  tasks  involving  identification  of  call 
signs,  operation  of  the  Fleet  Broadcast  (five  items),  operation  of  two 
teletype  terminals  and  operation  of  a receiver.  For  the  most  part  these 
are  procedures  and  equipment  that  are  used  primarily  aboard  ship.  Shore- 
based  personnel  now  do  receive  much  less  training  emphasis  on  these 
items  under  the  new  tracking  system  in  Radioman  "A"  School. 

Personnel  at  shore  duty  stations  classified  tasks  as  undertrained 
which  dealt  with  reading  of  perforated  teletype  tape,  operation  of  the 
Autodin  terminal,  conduct  of  emergency  destruction  of  classified  material 
and  five  other  items  related  to  security  practices.  Reading  perforated 
tape  and  Autodin  are  items  in  the  new  Radioman  "A"  School  curriculum 
that  were  not  included  in  the  training  of  the  persons  in  our  sample. 

The  tasks  related  to  security  practices  are  especially  critical  for 
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shore  installations  which  usually  are  in  foreign  countries.  Undertraining 
is  probably  the  case  for  these  tasks,  but  since  the  procedures  are 
generally  specific  to  the  particular  units,  more  on-the-job  training, 
not  more  school  training,  is  indicated. 

Items  indicated  to  be  overtrained  for  shipboard  personnel  included 
the  logging  of  outgoing  messages;  identification  of  duplicate  messages, 
special  messages  and  readdressed  messages;  distinguishing  between  ship 
and  shore  teletype  circuits;  and  distinguishing  among  the  three  major 
components  of  a shore  communication  station.  Also  included  were  tasks 
related  to  placing  tapes  in  backlog  bins,  operating  a particular  reper- 
forator, and  painting  antennas.  All  of  these  "overtrained"  tasks  except 
painting  antennas  are  primarily  shore-station  functions  which  now  do 
receive  much  less  emphasis  in  Radioman  "A"  School  for  ship-bound  trainees. 
Antenna  painting  did  not  receive  much  attention  in  "A"  School  in  the 
past  and  still  does  not.  The  overtraining  indicated  by  the  Training 
Adequacy  Index  may  have  been  due  to  the  simplicity  of  the  task. 

Undertraining  was  indicated  by  the  index  for  shipboard  personnel 
for  tasks  of  checking  the  accuracy  of  prepared  tape,  identification  of 
incorrect  Naval  Activity  Short  Titles,  activating  crypto  equipment,  and 
use  of  counter  measures  and  other  procedures  related  to  enemy  jamming. 

Short  Title  identification  is  a new  feature  of  Radioman  "A"  School  that 
was  not  included  in  the  training  of  our  sample.  More  emphasis  is  also 
now  given  to  operation  of  crypto  equipment  for  shipboard  personnel. 

However,  jamming  procedures  receive  less  emphasis  in  Radioman  "A"  School 
now  than  in  the  past  and  undertraining  indicated  by  the  index  may  be  a 
"false  alarm." 

In  summary,  the  derived  index  provides  generally  good  data  on  the 
training  problems  known  to  exist  in  the  old  Radioman  "A"  School.  Where 
it  fails,  it  is  usually  because  the  overtraining  indicated  is  unavoidable 
and  not  a school  problem,  or  because  the  index  does  not  distinguish 
between  undertraining  that  is  a school  problem  and  undertraining  that 
must  be  corrected  by  on-the-job  training.  The  moderate  success  of  this 
derived  Training  Adequacy  Index  speaks  well  of  the  reliability  and 
accuracy  of  the  Criticality  and  Performance  ratings  from  which  the  index 
was  derived. 

OPEN-ENDED  QUESTIONS 

For  all  instruments  compared  in  the  study,  open-ended  questions 
were  used  to  identify  missing  objectives  in  the  Radioman  "A"  School 
curriculum.  Six  different  categories  of  missing  objectives  were  mentioned; 
(1)  use  of  publications,  (2)  perforated  tape  reading,  (3)  radiotelegraph 
procedures  (International  Morse  Code),  (4)  NAVCOMPARS  (computer  communica- 
tion procedures),  (5)  satellite  communications,  and  (6)  voice  communications. 
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The  percentages  of  persons  mentioning  these  items  for  the  different 
instruments  and  for  different  respondent  groups  are  presented  in  table 
12.  Three  of  the  six  categories  of  missing  objectives  have  been  incorpor- 
ated in  the  new  Radioman  "A"  School  curriculum  and  tentative  plans  are 
being  made  to  include  a fourth  (voice  communications).  Discussions  with 
school  personnel  indicated  that  use  of  publications  may  also  be  a bona- 
fide  missing  objective  although,  until  we  reported  our  results,  it  was 
not  considered  for  inclusion  in  the  curriculum.  One  scale  rating  which 
appears  to  be  a "false  alarm"  by  the  questionnaire  instrument  may,  in 
fact,  not  be.  Radiotelegraph  procedures  are  taught  in  the  more  advanced 
conmunication  schools  and  presumably  are  related  to  the  jobs  of  higher 
rated  personnel.  Since  these  procedures  were  targeted  as  missing  training, 
it  indicates  that  some  "A"  School  graduates  are  being  placed  in  work 
situations  requiring  that  skill. 

A high  percentage  of  persons  receiving  structured  interviews  indicated 
the  task  of  reading  perforated  teletype  tape  as  a missing  training 
objective.  These  responses  were  mainly  from  personnel  at  Naval  Communica- 
tion Station  Puerto  Rico  where  this  task  is  frequently  performed.  This 
item  was  largely  responsible  for  the  greater  average  number  of  missing 
objective  comments  for  the  structured  interview  group.  However,  none  of 
the  differences  among  instruments  was  significant  nor  was  the  difference 
between  supervisors  and  trainees. 

The  short  questionnaire  depended  on  open-ended  questions  to  identify 
specific  areas  of  undertraining  and  overtraining.  Twenty  different 
tasks  received  mention  as  being  undertrained.  The  most  frequently 
mentioned  item  was  crypto  equipment  with  15  percent  of  respondents  on 
the  short  questionnaire  reporting  it  as  undertrained.  The  least  fre- 
quently mentioned  item  was  first  aid  which  was  mentioned  by  only  one 
percent  of  the  short  form  respondents.  Tasks  mentioned  by  five  percent 
or  more  of  respondents  typically  were  indicated  by  other  sources  to  be 
tasks  that  were  undertrained.  These  sources  included  the  changes  in  the 
old  curriculum  for  the  task  and  also  data  from  the  derived  Training 
Adequacy  Index  discussed  previously.  When  an  item  was  mentioned  by 
fewer  than  five  percent  of  respondents,  the  other  sources  usually  indi- 
cated adequate  training  or  even  overtraining  for  the  item.  First  aid 
was  one  of  the  areas  that  the  derived  Training  Adequacy  Index  indicated 
to  be  undertrained,  yet  only  one  percent  of  respondents  mentioned  this 
item.  It  is  not  clear  whether  this  reflects  on  the  open-ended  question 
as  a source  of  information  on  undertraining,  on  the  derived  Training 
Adequacy  Index,  or  on  both. 

Significantly,  fewer  comments  were  provided  about  overtraining  than 
were  made  about  undertraining.  The  only  area  mentioned  as  overtrained 
by  more  than  three  respondents  was  shipboard  equipment  and  systems. 

Since  practically  all  of  these  overtraining  comments  came  from  shore 
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TABLE  12.  PERCENTAGE  OF  RESPONDENTS  INDICATING  MISSING 

OBJECTIVES  ON  OPEN-ENDED  QUESTIONS  FOR  DIFFERENT 
INSTRUMENTS  AND  GROUPS 
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installations,  these  comments  appear  to  be  valid.  Again,  only  a small 
percentage  (eight)  mentioned  this  item,  but  it  is  indicative  of  the 
sensitivity  of  the  instrument. 

FOLLOW-UP  LETTERS  TO  RESPONDENTS  AND  NON-RESPONDENTS 

To  obtain  opinions  from  Fleet  personnel  regarding  this  particular 
feedback  effort  and  about  the  process  of  obtaining  feedback  information 
by  the  schools,  follow-up  letters  were  mailed  to  the  1026  trainees  and 
supervisors  who  had  not  returned  the  questionnaire  and  to  the  total 
sample  of  600  trainees  and  supervisors  who  had  responded.  Only  145 
letters  to  non-respondents  were  returned.  An  analysis  of  the  reasons 
given  for  not  returning  the  feedback  materials  indicated  that  the  Radio- 
men considered  the  materials  too  long  (15  percent  of  returned  letters, 
mostly  from  trainees),  did  not  receive  the  materials  (18  percent),  lost 
the  materials  (10  percent),  did  not  feel  qualified  to  evaluate  "A" 

School  (11  percent),  did  not  understand  the  questionnaire  (5  percent), 
were  transferred  (6  percent),  and  did  not  use  Radioman  "A"  School  training 
(9  percent).  It  is  interesting  to  note  that  59  percent  of  those  reporting 
that  they  either  did  not  receive  or  lost  the  materials  were  mailed  the 
long  questionnaire.  This  is  significantly  different  than  the  33  percent 
that  would  be  expected  by  chance  (z  = 6.1,  p<£.001). 

Of  the  600  follow-up  letters  mailed  to  respondents,  175  were  returned. 
A number  of  trainees  receiving  the  materials  within  the  time  frame  of  10 
or  more  months  following  graduation  from  school  indicated  difficulty  in 
remembering  the  school  curricula  and  generally  concurred  that  the  evalua- 
tion should  be  conducted  sooner  after  graduation  (23  percent).  In 
addition,  27  percent  of  the  respondents  felt  that  the  evaluation  forms 
should  be  tailored  to  the  tasks  required  at  ship  and  shore  duty  stations. 
Some  supervisors  (5  percent)  expressed  doubt  as  to  their  ability  to 
evaluate  the  trainee  based  on  the  "A"  School  curriculum  or  to  remember 
trainee  performance  upon  arrival  at  the  job  site. 

The  remaining  trainees  and  supervisors  indicated  no  problems  in 
completing  the  materials.  A number  of  personnel  indicated  their  pleasure 
at  being  asked  for  their  opinions  and  complimented  the  format  (11  percent). 
Some  respondents  indicated  a desire  to  receive  feedback  on  the  results 
of  this  study  and  future  evaluations. 

A small  number  of  respondents  indicated  that  they  did  not  appreciate 
the  criticality  of  the  return  of  evaluation  materials  until  they  received 
the  follow-up  inquiry.  Such  comments  support  the  importance  of  an 
expectation  letter,  in  order  to  insure  that  the  respondent  understands 
the  value  of  his  input  to  the  training  cycle. 
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SUMMARY  OF  RESULTS 
RETURN  RATE  DATA. 

Return  rate  for  supervisors  was  almost  double  that  for  trainees 
(59.3  percent  vs.  31.1  percent). 

. Return  rate  following  reminder  letters  was  greater  for  super- 
visors than  trainees  (21.6  percent  vs.  5.9  percent). 

. Return  rate  for  the  short  questionnaire  was  greater  than  for 

the  long  questionnaire  or  card  sort  (48.9  percent,  37.3  percent, 
and  38.2  percent,  respectively). 

Time  until  return  of  questionnaire  was  longer  for  the  long 
form  than  for  the  short  form  and  card  sort  (38.8  days,  30.7 
days,  and  33.2  days,  respectively). 

. For  trainees,  the  return  rate  for  the  short  form  was  greater 

than  for  the  long  form  or  card  sort  (40.0  percent,  25.3  percent, 
and  27.7  percent,  respectively). 

The  group  4 to  6 months  from  graduation  yielded  the  lowest 
return  rate  for  the  time  from  graduation  groups  (4  to  6,  35.3 
percent;  7 to  9,  41.2  percent;  10  to  12,  45.5  percent;  greater 
than  12,  43.8  percent). 

. Trainees  graduating  in  the  Upper  third  and  Middle  third  of 
their  classes  had  a higher  return  rate  than  the  Lower  third 
(35.1  percent,  33.3  percent,  and  23.9  percent,  respectively). 

The  group  receiving  "Expectation  Letters"  had  a higher  return 
rate  than  those  who  did  not  receive  them  (44.4  percent  vs. 

39.0  percent). 

RATING  SCALE  DATA. 

Correlations  between  rating  scales  were  higher  for  short 
forms: 


Short  Form 

Long  Form 

Variable 

.93 

.70 

Frequency  with  Criticality 

.97 

.84 

Frequency  with  Performance 

.94 

.70 

Criticality  with  Performance 
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Higher  ratings  were  given  for  shipboard  personnel  than  for 
shore-based  personnel  on  both  Frequency  and  Criticality  of 
Task  scales  (due  largely  to  the  inclusion  of  more  tasks  specific 
to  shipboard  duties  on  the  instruments). 

Average  ratings  for  the  Performance  of  Task  Upon  Arrival  From 
School  scale  were  nearly  identical  for  shipboard  and  shore- 
based  personnel . 

Correlations  between  shipboard  and  shore-based  personnel  for 
the  Frequency,  Criticality  and  Performance  scales  were  .72, 

.70,  and  .88,  respectively. 

Overall  average  ratings  for  trainees  and  supervisors  were 
nearly  identical  on  the  Frequency  and  Criticality  scales. 

Trainees  rated  their  own  Performance  higher  than  supervisors 
rated  this  performance,  but  the  basic  pattern  of  ratings 
across  tasks  was  similar.  The  correlations  between  the  Perform- 
ance ratings  of  the  two  groups  was  greater  than  .80  for  both 
the  long  form  and  card  sort  and  greater  than  .90  for  the  short 
questionnaire. 

Trainees  graduating  in  the  Upper  third  of  their  classes  and 
their  supervisors  rated  trainee  Performance  higher  than  graduates 
(and  their  supervisors)  from  the  Middle  and  Lower  groups. 

The  correlation  for  average  Performance  ratings  on  the  short 
form  between  normal  and  reversed  item  formats  was  .94. 

The  correlation  for  average  Performance  ratings  on  the  long 
form  between  normal  and  reversed  item  formats  was  .89 

Supervisor  ratings  of  Frequency  were  more  accurate  than  those 
of  trainees. 

Average  Frequency  ratings  were  more  accurate  under  the  joint 
administration  condition  than  under  the  single  administration 
condition. 

Trainees  in  the  lower  third  of  their  graduating  classes  were 
less  accurate  in  their  Frequency  ratings  than  trainees  in  the 
Upper  and  Middle  thirds. 

The  group  4 to  6 months  from  graduation  gave  more  accurate 
Frequency  ratings  than  other  groups. 

The  group  10  to  12  months  from  graduation  gave  less  accurate 
Frequency  ratings  than  other  groups. 
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A Training  Adequacy  Index  based  on  Criticality  of  Task  and 
Performance  of  Task  Upon  Arrival  From  School,  accurately 
identified  many  tasks  which  were  previously  known  to  be  over- 
trained or  undertrained. 

OTHER  FINDINGS. 

Open-ended  questions  identified  training  objectives  that  were 
missing  from  the  old  curriculum  that  have  been  included  in  the 
new  curriculum. 

Follow-up  inquiries  elicited  the  following  major  information: 

59  percent  of  those  reporting  loss  or  non-receipt  of  question- 
naires were  mailed  the  long  form.  Only  33  percent  would  be 
expected  by  chance  and  the  difference  was  highly  significant. 

Of  those  who  expressed  difficulty  completing  the  short  form, 

23  percent  expressed  an  inability  to  remember  the  school 
curriculum  as  a reference  point. 

A number  of  respondents  expressed  a desire  that  future  feedback 
questionnaires  be  specific  to  either  ship  or  shore  duty 
requirements  (27  percent). 
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SECTION  IV 


DISCUSSION  AND  CONCLUSIONS 

The  most  compelling  aspect  of  the  results  of  this  study  was  the 
consistent  patterns  of  average  item  ratings  that  were  found  on  the 
scales  of  Frequency  of  Task,  Criticality  of  Task,  and  Performance  of 
Task  Upon  Arrival  From  School.  Similar  patterns  were  produced  by  each 
group  of  respondents  and  with  each  administration  procedure.  None  of 
the  product-moment  correlations  calculated  between  different  groups  and 
between  administration  procedures  was  less  than  .70  and  many  were  .90  or 
more.  This  high  reproducibility  of  average  item  ratings  occurred  even 
for  groups  in  which  the  number  of  respondents  was  50  or  less. 

Not  only  were  the  average  ratings  for  tasks  on  the  different  scales 
highly  reliable,  but  they  corresponded  to  the  differences  that  were 
known  to  exist  prior  to  questionnaire  administration  in  frequency, 
criticality,  and  trainee  performance  for  these  tasks.  The  large  ship- 
shore  differences  in  task  frequency,  which  were  the  basis  for  the  recent 
major  curriculum  modification  of  Radioman  "A"  School,  are  accurately 
reflected  in  the  Frequency  ratings.  Shipboard  tasks  were  indicated  as 
"Never  Performed"  by  nearly  all  shore-based  personnel  and,  similarly, 
tasks  known  never  to  be  performed  by  shipboard  personnel  were  indicated 
as  "Never  Performed"  by  nearly  all  shipboard  respondents.  This  result 
indicates  that,  with  very  few  exceptions,  respondents  completed  the 
questionnaires  with  much  care. 


Performance  ratings,  like  the  Frequency  ratings,  reflected  the 
real-world  situation.  The  ability  to  read  perforated  teletype  tapes  is 
important  for  many  shore-based  personnel,  but  this  task  was  not  yet 
included  in  the  curriculum  of  those  persons  interrogated  in  our  study. 
Performance  of  Task  Upon  Arrival  From  School  ratings  were  found  to  be 
very  low  for  this  task.  Items  related  to  International  Morse  Code  use 
also  were  given  very  low  Performance  ratings  and  this,  too,  was  expected 
since  Code  training  was  not  (and  is  not)  included  in  Radioman  "A"  School. 
Highly  significant  differences  appeared  between  items  on  the  Criticality 
scale  and  some  of  the  highest  Criticality  ratings  occurred  for  tasks 
dealing  with  first  aid  and  with  security  practices.  These  ratings  on 
this  scale  appear  to  be  highly  valid  responses  as  well. 

Differences  that  existed  for  average  ratings  among  different  groups 
of  the  study  also  indicate  the  validity  of  the  rating  scale  data.  A 
direct  relationship  appeared  between  ratings  of  Performance  of  Task  Upon 
Arrival  From  School  and  the  three  Class  Standing  groups.  Performance  of 
Task  Upon  Arrival  From  School  was  rated  highest  for  the  upper  one-third 
and  was  rated  lowest  for  the  lower  one-third.  The  frequency  of  task 


51 


TAE6  Report  No.  19 


performance  typically  increases  as  trainees  gain  more  experience  and 
Frequency  ratings  showed  significant  increases  for  groups  with  longer 
time  from  graduation.  This  finding  also  indicates  the  validity  of  these 
average  ratings. 

The  high  reliability  and  high  validity  of  the  present  rating  scale 
data  indicate  that  mail -out  questionnaires  can  provide  excellent  data 
that  can  serve  as  a means  for  identification  of  training  problems.  This 
finding  is  contrary  to  an  expectation  that  developed  early  in  the  study 
that  only  expensive  face-to-face  interviews  with  personnel  of  operational 
units  could  provide  effective  data  for  these  purposes.  This  bias  was 
generated  by  a number  of  published  and  unpublished  studies  which  reported 
unsuccessful  results  using  mail-out  questionnaires  (e.g.,  TAE6  Report 
No.  12-1,  1973).  With  minor  modifications,  questionnaire  format  and 
administration  procedures  used  in  this  study  appear  to  be  applicable  to 
most,  if  not  all.  Navy  training  situations.  The  questionnaire  and 
procedures  are  described  in  Section  V of  this  report  and  are  described 
in  more  detail  in  a procedure  manual  which  is  being  published  separately 
as  TAEG  Feedback  Manual  (June  1975). 

The  structured  interview  technique,  which  elicited  our  initial 
faith,  was  also  examined  in  the  present  study  but,  as  reported  earlier, 
no  basic  differences  between  the  structured  interview  and  long  question- 
naire were  found  in  the  ratings  of  job  tasks  or  in  the  responses  to  the 
open-ended  questions.  In  fact,  the  highest  correlation  found  in  the 
study  was  between  Performance  ratings  on  the  long  questionnaire  and 
Performance  ratings  on  the  structured  interview  (.94).  Variability  of 
item  ratings  also  did  not  differ  between  ratings  obtained  from  structured 
interviews  and  those  obtained  in  the  mail-out  questionnaire  (long  form). 
The  similar  variability  of  item  ratings  indicates  that  one  returned 
questionnaire  is  equivalent  to  one  structured  interview.  Although  the 
structured  interviews  enabled  the  gathering  of  information  that  was  not 
specifically  requested  in  the  questionnaire,  this  information  could  have 
been  obtained  by  mailing  a second  questionnaire.  The  structured  inter- 
view should  be  used  during  the  development  of  the  mail-out  instrument. 
Interviews  could  be  conducted  with  training  personnel,  especially  those 
personnel  recently  assigned  from  operational  units.  This  will  allow 
identification  of  questionnaire  ambiguities,  and  identify  additional 
useful  information  that  should  be  requested  in  the  questionnaire  (see 
section  V) . 

As  was  noted  earlier,  statistically  significant  differences  were 
found  in  the  accuracy  of  ratings  for  different  respondent  groups  and  for 
different  procedural  conditions  in  the  present  study.  However,  despite 
these  differences,  very  high  correlations  existed  between  ratings  of 
more  accurate  and  less  accurate  groups.  For  this  reason,  only  the 
largest  difference  in  rating  accuracy,  which  appeared  between  supervisors 
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and  trainees,  would  influence  these  procedures  for  obtaining  post 
formal  training  feedback.  Trainee/supervisor  differences  in  ratings  for 
an  item  often  occurred  due  to  ambiguous  task  descriptions  or  because  a 
piece  of  equipment  was  identified  by  model  number  (e.g.,  AN/SRC-20)  or 
other  name  that  conveyed  little  information  about  it  or  its  use.  Under 
these  circumstances,  supervisors  were  more  apt  than  trainees  to  give 
Frequency  ratings  that  were  consonant  with  the  known  frequency  of  perform- 
ance of  the  task.  This  result  implies  that  more  weight  should  be  given 
to  supervisor  ratings  on  task  items  when  these  ratings  differ  from  those 
of  the  trainee.  The  appearance  of  trainee/supervisor  differences  for 
ratings  on  a task  may  also  indicate  that  the  item  requires  revision 
prior  to  the  next  series  of  mail-out  questionnaires. 

The  large  differences  in  return  rate  among  the  different  types  of 
mail-out  instruments  were  directly  related  to  the  length  of  the  instru- 
ment. Follow-up  letters  sent  to  persons  who  did  not  return  the  long 
questionnaire  often  were  returned  with  the  "Questionnaire  was  too  long" 
category  checked.  The  advantage  of  the  short  questionnaire  was  particu- 
larly striking  for  trainees.  More  than  one  and  one-half  times  as  many 
short  forms  than  long  forms  were  returned  by  this  group.  For  supervisors, 
only  10  percent  more  short  forms  than  long  forms  were  returned.  These 
return  rate  differences  between  long  and  short  instruments  were  closely 
paralleled  by  differences  in  time  to  return  these  instruments,  with  more 
time  required  for  return  of  the  long  form. 

Although  the  return  rate  was  higher  for  the  short  questionnaire, 
much  less  information  about  training  problems  resulted  from  this  instrument. 
Often,  the  general  objectives  listed  in  the  short  form  were  rated  as 
having  "Adequate"  performance  whereas  on  the  long  form  some  enabling 
objectives  subsumed  under  the  general  objective  were  rated  "Substantially 
Inadequate."  To  a limited  extent,  the  lack  of  specificity  of  the  short 
questionnaire  was  compensated  for  by  open-ended  questions  which  were 
included  to  uncover  specific  problems.  However,  such  a small  fraction 
of  respondents  mentioned  any  particular  problem  that  it  is  difficult  to 
assess  the  extent  of  the  training  problem.  With  the  long  questionnaire, 
the  specific  item  is  rated  by  all  respondents  and  the  training  problem 
appears  prominently  against  the  background  of  non-problem  items. 

Considering  the  difficulties  in  identifying  specific  problems  with 
the  short  questionnaire,  it  appears  that  the  factor  of  nine  that  was  the 
ratio  of  items  between  the  long  and  short  forms  (134/15)  also  described 
the  ratio  of  information  provided  about  training  problems.  Other  evidence 
for  the  poor  information  source  of  the  short  questionnaire  was  the  near 
perfect  correlation  between  scales  on  the  short  forms  (see  table  6). 

This  indicated  that  no  more  information  was  communicated  by  ratings  on 
all  three  scales  than  was  communicated  by  ratings  on  any  one  of  the 
scales. 
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No  differences  appeared  between  the  normal -item-order  long  question- 
naire and  the  reversed-i tern-order  long  questionnaire.  In  fact,  the 
correlation  of  .90  between  Performance  ratings  on  these  two  forms  was 
one  of  the  highest  correlations  that  appeared  in  the  study  for  the 
Performance  scale  of  the  long  questionnaire.  This  indicates  that  question- 
naire accuracy  was  constant  or  nearly  constant  throughout  all  134  items, 
and  that  even  longer  questionnaires  can  be  used,  if  necessary,  to  provide 
training  feedback. 

As  was  noted  in  the  previous  section,  the  differences  in  return 
rate  between  trainees  and  supervisors  were  large  with  nearly  twice  as 
many  questionnaires  returned  by  supervisors  as  by  trainees.  The  higher 
return  rate  of  the  supervisor  may  reflect  the  fact  that  poor  training 
places  a large  burden  on  him.  The  readiness  of  his  unit  is  reduced  by  a 
poorly  trained  person  and  he  must  take  much  of  his  time  to  bring  this 
person  "up-to-speed. " The  higher  return  rate  may  also  be  due  to  the 
fact  that  the  supervisor  is  more  accustomed  to  administrative  tasks  and 
that  he  has  more  time  than  the  trainee  for  "paperwork."  Despite  this 
large  difference  in  return  rate  between  trainees  and  supervisors,  strong 
reasons  exist  for  including  both  the  trainee  and  supervisor  in  post 
formal  training  feedback.  The  trainee  has  a more  up-to-date  knowledge 
of  the  existing  training  in  school  and  has  direct  knowledge  of  his  own 
performance  capabilities  for  job  tasks.  In  theory,  at  least,  he  can 
provide  more  accurate  data  than  the  supervisor  on  certain  training 
problems.  Another  reason  for  interrogating  both  groups  is  that  super- 
visory personnel  are  often  already  burdened  with  a great  deal  of  paper- 
work, and  sharing  the  feedback  load  between  supervisors  and  trainees 
will  relieve  some  of  the  supervisor's  workload.  The  post  formal  training 
feedback  task  will  thus  be  less  onerous  for  Fleet  personnel. 

Since  the  "expectation"  factor  improved  the  return  rate  for  trainees, 
the  practice  of  mailing  advance  letters  to  trainees  should  be  followed 
in  any  initial  effort  for  obtaining  feedback  information.  However,  a 
better  procedure  for  the  continuing  feedback  effort  would  be  to  discuss 
the  feedback  function  with  the  student  prior  to  his  departure  from 
school.  The  materials  could  also  be  shown  and  explained  to  him  at  that 
time,  but  the  materials  should  not  be  sent  with  him  to  the  duty  station. 
Discussions  with  training  personnel  indicate  that  sending  the  materials 
with  the  trainee  to  his  new  duty  station  has  often  resulted  in  very  low 
return  rates.  For  this  reason,  the  evaluation  materials  should  be  sent 
at  the  time  they  are  to  be  completed.  Since  the  return  rate  from  super- 
visors was  increased  only  slightly  by  advance  notice,  the  "expectation" 
letter  for  supervisors  can  be  omitted. 

Joint  administration  (both  the  trainee  and  his  supervisor)  produced 
a higher  return  rate  than  single  administration  which  was  accompanied  by 
higher  accuracy  and  more  comments.  Although  these  features  augur  well 
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for  a policy  of  joint  administration,  they  are  countered  by  information 
gleaned  from  follow-up  letters  to  trainees  responding  in  this  joint 
administration  mode.  In  several  instances,  there  were  clear  indications 
that  the  trainee  had  actually  filled  out  both  his  questionnaire  and  that 
of  his  supervisor.  The  fact  that  some  trainees  revealed  this  information 
could  mean  that  numerous  trainees  were  required  to  fill  out  the  supervisor's 
form.  For  this  reason,  either  the  supervisor  or  the  trainee  should  be 
contacted,  but  both  should  not  be  contacted  unless  the  number  of  graduates 
in  a school  is  small  and  both  are  needed  to  provide  reliable  information. 

If  the  trainee  fills  out  the  supervisor's  form  in  such  single  administration 
(or  vice  versa),  duplicate  information  will  not  be  collected. 

Return  rate  differences  were  small  for  groups  who  had  different 
time  periods  from  graduation  prior  to  the  mailing  of  questionnaires; 
therefore,  this  measure  provides  little  basis  for  selecting  one  time 
period  over  another.  However,  follow-up  letters  to  persons  who  were 
more  than  nine  months  from  graduation  prior  to  interrogation  often 
indicated  that  the  respondent  believed  he  should  have  received  the 
materials  earlier.  Early  interrogation  also  allows  for  earlier  identi- 
fication and  correction  of  problems.  Some  minimum  time,  however,  is 
required  on  the  job  to  enable  the  trainee  to  evaluate  his  performance; 
to  determine  the  adequacy  of  his  training  for  this  performance;  and  for 
the  supervisor  to  evaluate  the  trainee's  performance.  Probably  three 
months  would  be  an  optimum  balance  of  these  different  factors.  However, 
in  the  Navy,  many  graduates  spend  the  first  three  months  of  their  tours 
mess-cooking  or  on  other  details  unrelated  to  their  training.  In  addition, 
leave  often  occurs  after  school  and  prior  to  assignment  to  the  duty 
station.  All  of  these  factors  taken  together  indicate  an  optimum  period 
of  between  five  and  seven  months  following  graduation  prior  to  interrogation. 

The  novel  and  easy  card-sort  response  required  for  the  mail -out 
card-sort  instrument  was  expected  to  produce  a high  rate  of  return; 
however,  the  data  did  not  support  this  expectation.  Although  slightly 
higher  than  the  return  rate  for  the  long  questionnaire,  the  difference 
in  return  rate  (1  percent)  was  too  small  to  argue  for  adoption  of  this 
instrument  and  procedure.  Even  though  the  card-sort  method  successfully 
produced  data  about  Trainee  Performance  Upon  Arrival  From  School,  which 
corresponded  closely  to  that  of  the  long  questionnaire  (r=.89),  it 
provided  only  partial  data  on  Task  Frequency  and  none  on  Task  Criticality. 

If  additional  scales  such  as  Task  Frequency  or  Task  Criticality  had  been 
included,  sorting  of  cards  would  have  become  a cumbersome  task  and  this 
would  negate  the  expected  advantage  of  the  easy  response.  Ambiguous 
task  statements  tended  to  cause  more  problems  with  this  instrument  than 
with  the  long  questionnaire.  This  suggests  that  the  response  of  card- 
sorting sometimes  occurs  without  enough  time  taken  for  analysis  and 
reflection.  For  these  reasons,  the  card-sort  technique  is  not  recommended 
for  general  post  formal  training  feedback  purposes. 
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Having  both  Criticality  of  Task  and  Performance  of  Task  Upon 
Arrival  From  School  scales  on  the  short  and  long  questionnaires  provided 
an  opportunity  to  calculate  a derived  Training  Adequacy  Index.  This 
involved  subtracting  the  Criticality  ratings  for  an  item  from  the  Perform- 
ance ratings  for  the  same  item.  Items  with  high  Performance  ratings  and 
low  Criticality  ratings  thus  received  large  values  and  items  with  low 
Performance  ratings  and  high  Criticality  ratings  received  small  values 
on  this  derived  scale  of  Training  Adequacy.  A similar  procedure  has 
been  used  by  Siegel,  Schultz,  and  Federman  (1961)  to  identify  overtraining 
and  undertraining  in  four  Navy  ratings.  When  this  procedure  was  applied 
to  data  from  the  long  questionnaire,  it  assisted  in  the  identification 
of  areas  of  known  undertraining  and,  to  a lesser  extent,  areas  of  known 
overtraining.  However,  in  structured  interviews,  items  which  had  the 
derived  characteristic  of  "undertraining"  often  did  not  elicit  a similar 
response  from  the  interviewee  when  he  was  directly  asked  if  more  school 
training  was  needed  for  the  task.  An  example  of  this  was  the  item 
"Conduct  emergency  destruction  of  classified  material"  which  persons 
interviewed  face-to-face  almost  unanimously  believed  was  best  learned  on 
the  job.  Conversely,  some  areas  where  the  derived  score  indicated 
overtraining, were  not  judged  as  overtrained  in  school  when  the  question 
was  asked  (e.g.,  "Changing  teletype  tapes  and  ribbons").  Since  the 
respondents  in  the  interviews  could  readily  express  opinions  on  training 
adequacy  for  a task,  and  since  these  opinions  were  often  largely  indepen- 
dent of  their  Criticality  and  Trainee  Performance  ratings,  it  is  important 
that  both  trainees  and  supervisors  be  asked  to  directly  rate  the  adequacy 
of  training  rather  than  to  rate  the  two  factors  of  Criticality  and 
Performance  of  Task  Upon  Arrival  From  School.  However,  retention  of  the 
Frequency  scale  is  recommended  to  aid  in  identifying  obsolete  equipment 
and  tasks.  The  relatively  easy  rating  of  Task  Frequency  may  also  serve 
the  function  of  getting  the  item  solidly  into  consideration  by  the 
respondent  prior  to  the  more  difficult  rating  of  Training  Adequacy.  We 
expect  that  the  Frequency  rating  will  not  inaccurately  bias  the  rating 
of  Training  Adequacy. 

Open-ended  questions  were  used  for  two  different  purposes  in  the 
instruments  administered.  For  both  long  and  short  questionnaires,  they 
were  included  to  elicit  new  material  that  should  be  included  in  the 
school  curriculum.  Our  recommended  instrument  and  procedure  also 
utilizes  an  open-ended  question  for  this  purpose  since  it  is  critical 
that  a way  be  provided  for  identifying  new  training  needs.  For  example, 
material  that  is  just  now  being  integrated  into  the  Radioman  "A"  School 
curriculum  was  mentioned  as  missing  from  the  curriculum  by  about  three 
percent  of  the  respondents.  This  implies  that  consideration  must  be 
given  to  ideas  for  new  training  if  only  a small  percentage  of  persons 
report  it.  The  other  purpose  of  open-ended  questions  was  to  get  at  the 
specific  problems  the  short  questionnaire  could  not  identify  with  the 
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rating  scales.  As  was  mentioned,  the  success  of  this  procedure  was 
limited  and  we  recommend  the  use  of  a long  form  which  mentions  the 
specific  situations,  thus  allowing  all  respondents  to  make  judgments  on 
possible  training  problems. 
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SECTION  V 

RECOMMENDED  FEEDBACK  INSTRUMENT  AND  PROCEDURES 

The  long  questionnaire  used  in  the  present  study  accurately  identified 
the  known  problems  of  the  old  Radioman  "A"  School  curriculum  and  provided 
almost  identical  data  to  that  obtained  from  more  expensive  face-to-face 
interviews.  For  these  reasons,  it  is  recommended  that  such  mail-out 
questionnaires,  based  on  specific  job  tasks,  be  adopted  throughout  the 
Navy  as  the  major  method  for  obtaining  post  formal  training  feedback. 

This  section  describes  the  recommended  instrument  and  procedures  which 
are  modified  forms  of  the  highly  successful  instrument  and  procedures 
used  in  the  present  study  and  described  earlier  in  this  report.  The 
modifications  make  the  instrument  even  more  effective  and  simplify  its 
administration  and  use. 

The  proposed  feedback/curriculum  revision  cycle  requires  18  months 
plus  the  duration  of  the  training  course.  In  addition,  initial  instrument 
development  may  take  from  one  to  five  weeks  with  the  longer  time  required 
if  job  task  statements  do  not  already  exist  and  the  existing  curriculum 
must  be  converted  to  descriptions  of  the  job  behaviors  for  which  the 
school  provides  training. 

A minimum  number  of  personnel  would  be  required  to  develop  these 
instruments  and  carry  out  these  procedures.  It  would  probably  not 
require  the  full  time  of  one  person  involved  with  curriculum  development 
except  during  the  initial  instrument  development  stage  and  the  data 
analysis  phase  which  occurs  after  all  questionnaires  have  been  returned. 
Typing,  printing,  and  clerical  services  will  be  required  for  brief 
periods.  Once  the  data  are  analyzed,  the  resultant  information  on 
training  problems  would  be  fed  into  the  normal  curriculum  revision 
process  and  the  person  who  develops  and  administers  the  instrument  would 
then  be  free  to  devote  full  time  to  this  revision  process. 

The  recommended  procedures  are  generally  straightforward  and  require 
no  particular  skills  or  training  of  the  person  assigned  to  carry  them 
out.  However,  if  this  person  enjoys  data  tabulation  and  manipulation, 
so  much  the  better.  A companion  report  (TAEG  Feedback  Manual , June 
1975)  is  written  for  use  by  relatively  unsophisticated  training  personnel 
to  enable  them  to  prepare  and  administer  the  instruments  and  to  analyze 
and  use  the  data. 

FORMAT  AND  PREPARATION  OF  FEEDBACK  QUESTIONNAIRE 

The  basic  form  which  will  make  up  the  bulk  of  the  questionnaire,  is 
the  next  to  last  page  of  each  of  the  questionnaires  presented  in  appendixes 
J and  K.  It  includes  brief  instructions,  space  for  eight  job  tasks,  and 
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two  scales  on  which  each  of  the  items  are  to  be  rated  by  circling  a 
number  from  one  to  five.  (If  optical  scoring  is  available,  numbered, 
dotted  parallel  lines  could  be  substituted  for  these  numbers.)  The 
"Frequency  of  Task"  scale  is  nearly  identical  to  that  used  in  the  present 
study.  No  respondent  reported  any  particular  difficulty  with  this 
scale.  However,  respondents  often  felt  it  difficult  to  rate  a task  on 
the  Criticality  and  Performance  scales  if  they  had  marked  the  task  as 
"Never  Performed"  on  the  Frequency  scale.  In  fact,  despite  instructions 
not  to  do  so,  they  often  skipped  these  other  scales  rather  than  make 
what  they  believed  to  be  inappropriate  estimates  about  the  criticality 
of  the  task  and  the  trainee's  ability  to  perform  the  task.  On  the  new 
form,  the  respondent  is  given  the  option  of  skipping  the  Adequacy  of 
School  Training  for  This  Task  scale  if  the  task  is  rated  as  "Never 
Performed. " 

The  Adequacy  of  School  Training  for  This  Task  scale  differs  from 
the  previous  instrument  which  asked  for  Criticality  of  Task  and  Perform- 
ance of  Task  Upon  Arrival  From  School.  As  was  mentioned  in  the  previous 
section,  it  is  expected  that  this  will  provide  more  useful  data  since 
structured  interviews  often  indicated  that  supervisors  and  trainees  made 
judgments  about  the  adequacy  of  school  training  which  were  relatively 
independent  of  their  ratings  of  Criticality  of  Task  and  Performance  of 
Task  Upon  Arrival  From  School. 

We  recommend  the  basic  job  task  form  be  used  without  change. 

Enough  copies  of  this  form  should  be  reproduced  to  accommodate  all  of 
the  specific  job  tasks  to  be  included  in  the  questionnaire.  These  job 
task  statements  should  be  written  in  terms  of  observable  behavior.  If  a 
job  task  analysis  has  been  conducted,  these  statements  will  already 
exist  and  preparation  of  the  main  body  of  the  questionnaire  can  be  done 
very  quickly.  Before  final  typing,  however,  it  is  essential  that  several 
persons  at  the  training  unit,  such  as  recent  school  graduates,  instructors 
and  recent  transfers  from  operational  units,  read  these  items  to  help 
identify  any  possible  problems  that  might  produce  misinterpretations  of 
the  task  statements.  We  suggest  that  these  persons  be  asked  to  rate 
Task  Frequency  and  Training  Adequacy  for  each  of  these  items  in  a structured 
interview  similar  to  that  which  was  conducted  in  the  present  study. 

If  job  task  statements  are  not  available,  the  preparer  of  the 
instrument  must  convert  areas  of  the  existing  curriculum  into  such  task 
statements.  This  conversion  of  a curriculum  to  task  statements  will  add 
much  time  to  questionnaire  development  and  will  require  even  more  structured 
interviews  with  personnel  for  the  purpose  of  editing  task  statements 
prior  to  typing  the  final  form  of  the  questionnaire. 

In  most  circumstances,  supervisors  will  receive  125  questionnaires 
similar  to  the  one  in  appendix  J,  and  another  200  questionnaires 
(appendix  K)  will  be  mailed  to  trainees.  Cover  letters  and  biographical 
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data  sheets  for  these  questionnaires  should  be  modified  from  the  forms 
in  the  appendixes  to  suit  the  needs  and  purposes  of  each  training  activity. 
One  critical  feature  that  should  be  retained  in  the  cover  letter  is  a 
deadline  for  return  of  the  instrument.  It  is  hoped  that  the  instructions 
and  open-ended  questions  can  be  reproduced  directly  from  these  appendixes 
without  modification. 

If  the  number  of  task  statements  exceeds  200,  two  different  question- 
naires should  be  made,  with  each  containing  half  of  the  statements. 

This  now  will  require  twice  as  many  persons  to  be  sampled  in  order  to 
obtain  information  about  the  course. 

ADMINISTRATION  PROCEDURES 

If  more  than  325  trainees  graduate  in  a six-month  period,  then  200 
of  these  should  be  selected  random! y from  the  graduation  rosters  of  the 
last  six  months  for  mailing  of  trainee  questionnaires  and  125  should  be 
randomly  selected  for  supervisor  questionnaires.  The  randomization  is 
important  since  it  will  minimize  biases  in  the  questionnaire  data  which 
might  arise  from  improper  sampling  of  different  ability  groups,  or 
different  duty  stations.  The  above  numbers  are  predicated  on  the  assump- 
tion that  less  than  200  task  statements  are  used  on  one  questionnaire. 

If  two  questionnaires  are  used,  twice  as  many  persons  (if  available) 
will  be  needed  in  each  of  the  above  groups. 

The  questionnaires  should  be  mailed  six  months  after  graduation, 
with  one  month  or  more  leeway.  The  supervisor  forms  should  be  mailed  to 
the  Commanding  Officer  at  the  trainee's  duty  station  with  instructions 
to  forward  the  form  to  the  "Supervisor  of  Seaman  (Name)."  The  trainee 
form  can  be  mailed  directly  to  the  trainee  at  his  duty  station.  Both 
letters  should  include  a self-addressed  envelope  for  return  of  the 
completed  questionnaire.  Two  weeks  prior  to  mailing  the  questionnaire 
to  a trainee,  the  trainee  should  be  sent  a letter  that  alerts  him  to  the 
imminent  arrival  of  the  questionnaire  and  its  purpose,  and  that  requests 
him  to  consider  the  adequacy  of  his  training  during  the  waiting  period. 

If  the  trainee  does  not  return  his  completed  form  within  one  month 
after  mailing,  a reminder  letter  (appendix  L)  should  be  mailed  directly 
to  him  and  not  to  his  Commanding  Officer.  No  reminder  should  be  sent  to 
the  supervisor  who  does  not  return  the  questionnaire  since  he  may  have 
recently  returned  one  for  another  graduate  (see  cover  letter  in  appendix 
J).  Only  the  supervisor  questionnaires  will  be  mailed  to  the  Commanding 
Officer  and  this  will  reduce  handling  of  materials  by  personnel  at 
operational  units.  Every  effort  should  be  made  to  minimize  the  burden 
that  post  formal  training  feedback  places  on  persons  in  supervisory  and 
command  positions  since  they  often  are  already  flooded  with  paperwork. 
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With  an  abundance  of  graduates,  it  would  be  possible  to  mail  more 
questionnaires  than  the  initial  200  to  trainees  and  125  to  supervisors. 
This  should  be  done  if  it  becomes  apparent  that  the  return  will  fall 
considerably  short  of  the  goal  of  75  trainee  forms  and  75  supervisor 
forms.  If  less  than  325  trainees  graduate  in  a six-month  period,  then 
at  least  some  forms  can  be  mailed  to  both  the  trainee  and  his  supervisor, 
thereby  increasing  the  number  of  returned  questionnaires.  With  less 
than  200  graduates,  all  trainees  and  all  supervisors  of  the  trainees 
could  be  sampled.  This  should  result  in  more  returned  questionnaires 
for  supervisors  than  for  trainees,  but  this  imbalance  is  less  important 
than  obtaining  the  150  returned  forms  needed  to  provide  reliable,  repre- 
sentative data  on  the  rating  scales.  If  the  school  has  a very  small 
number  of  graduates,  the  period  of  interrogation  can  be  extended  from 
six  months  to  a year  or  longer  in  order  to  build  up  a satisfactory 
number  of  returned  questionnaires.  In  addition,  more  intensive  follow- 
up procedures  (e.g.,  phone  calls,  a second  questionnaire,  contact  of 
supervisors)  can  be  used  to  obtain  returned  questionnaires. 

The  sample  of  graduates  should  be  selected  from  graduating  classes 
over  a six-month  period.  One  reason  is  that  temporary  school  problems 
will  not  have  as  much  influence  in  the  results.  Also,  over  a short 
period  of  time  an  unrepresentative  sampling  of  duty  stations  could 
easily  occur.  Another  advantage  of  a six-month  interrogation  period  is 
that  it  greatly  reduces  the  workload  involved  with  mailing  question- 
naires and  recording  the  returned  data.  Only  about  eight  expectation 
letters,  13  questionnaires  and  five  follow-up  letters  would  need  to  be 
mailed  each  week  and  only  data  from  about  six  returned  questionnaires 
recorded.  This  would  require  a small  amount  of  time  and  allow  opportun- 
ity for  checking  the  outgoing  materials  and  insuring  that  they  are  free 
of  defects. 

PROCEDURES  FOR  ANALYSIS  AND  UTILIZATION  OF  QUESTIONNAIRE  DATA 

A notebook  should  be  prepared  with  a page  for  each  item  on  the 
questionnaire.  This  page  could  take  the  form  of  the  example  in  appendix 
M,  which  has  space  for  maintaining  separate  records  for  Trainee  Frequency 
and  Training  Adequacy  and  Supervisor  Frequency  and  Training  Adequacy. 

When  a questionnaire  is  returned,  an  identification  number  should  be 
assigned  to  it.  This  identification  number  can  be  written  on  each  item 
page  in  the  columns  that  correspond  to  that  person's  ratings  on  Frequency 
and  on  Adequacy.  Any  comments  which  are  written  regarding  training  for 
an  item  should  be  written  on  the  back  of  the  particular  notebook  page 
for  that  item  and  the  identification  number  included  to  identify  the 
origin  of  the  conment.  When  a returned  questionnaire  has  a response  to 
the  open-ended  question  asking  for  new  material  that  should  be  included 
in  the  curriculum,  a new  page  for  the  notebook  should  be  made  on  which 
this  response  is  written  along  with  the  identification  number  of  its 
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author.  A separate  page  should  be  added  for  each  new  item  mentioned. 

As  future  questionnaires  arrive  that  also  mention  this  item,  their 
identification  numbers  should  be  added  to  this  page. 

After  all  questionnaires  have  been  mailed  and  follow-up  procedures 
fail  to  produce  more  returns,  and  if  the  number  returned  are  at  least  50 
for  the  trainees  and  50  for  the  supervisors  (if  less,  mail  more  question- 
naires), the  data  analysis  may  begin.  Separate  statistical  means  should 
be  calculated  for  each  item  for  trainee  ratings  of  Frequency,  trainee 
ratings  of  Training  Adequacy,  supervisor  ratings  of  Frequency,  and 
supervisor  ratings  of  Training  Adequacy.  The  numbers  of  Training  Adequacy 
ratings  will  differ  somewhat  across  pages  (items)  because  of  the  option 
respondents  have  of  skipping  that  scale.  It  is  important  to  calculate 
the  mean  with  the  actual  number  of  ratings  given  for  the  item  and  not 
with  the  total  number  of  trainees  or  supervisors  who  returned  question- 
naires. 

In  addition  to  these  four  means,  an  additional  mean  should  be 
calculated  for  each  item  sheet  which  is  the  average  of  the  average 
rating  of  Training  Adequacy  for  all  trainees  and  of  the  average  rating 
of  Training  Adequacy  for  all  supervisors.  After  this  "overall  rating  of 
Training  Adequacy"  has  been  calculated  for  all  items,  the  pages  for  the 
items  should  be  reordered  with  high  values  of  this  average  at  one  end 
and  low  values  at  the  other. 

The  10  percent  of  items  at  each  end  of  this  reordered  stack  are 
prime  candidates  for  an  investigation  of  training  and  possible  curriculum 
revisions.  Unless  other  factors  argue  very  strongly  against  it,  those 
items  which  are  closest  to  the  "task  requires  much  more  emphasis  in 
school"  end  of  the  Adequacy  of  School  Training  scale  should  be  allotted 
more  training  emphasis.  Similarly,  the  items  which  are  closest  to  the 
"greatly  reduce  or  eliminate  training  for  this  task"  end  of  the  scale 
should  have  the  time  and  other  resources  given  to  their  training  reduced. 
When  reduced  training  emphasis  is  recommended  and  ratings  of  Task  Frequency 
indicate  a very  low  rate  of  performance,  the  situation  is  probably  one 
of  irrelevant  training.  More  drastic  curriculum  changes  may  be  called 
for  than  in  the  overtraining  condition  where  reduced  training  is  recommended 
but  frequency  of  performance  of  the  task  is  moderate  or  high. 

Many  factors  operate  to  bring  average  ratings  of  Training  Adequacy 
toward  the  midpoint  of  the  scale  and  average  ratings  of  highest  and 
lowest  items  may  differ  by  only  a little  more  than  one  point.  This 
should  not  be  used  as  an  excuse  to  refrain  from  curriculum  revision. 

The  standard  error  of  an  average  of  150  item  ratings  will  be  less  than 
.1.  If  scales  were  marked  randomly,  the  chances  of  a difference  of  one 
scale  unit  or  more  between  highest  and  lowest  items  would  be  less  than 
one  in  one  thousand.  Thus,  the  differences  in  average  Training  Adequacy 
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between  the  top  and  bottom  10  percent  of  items  can  be  assumed  to  be  real 
differences  and  not  chance  occurrences.  Unless  the  school  is  an  exceptional 
one,  most  items  falling  within  the  10  percent  margins  can  and  should  be 
acted  upon.  If  personnel  and  time  are  not  available  to  correct  training 
for  all  of  the  top  and  bottom  10  percent  of  these  items,  those  with  the 
most  extreme  average  ratings  of  Training  Adequacy  should  be  attended  to 
first. 

A factor  that  might  countermand  the  recommendations  of  these  average 
Training  Adequacy  ratings  would  be  if  a very  large  percentage  (perhaps 
70  percent)  of  the  total  sample  skipped  the  Training  Adequacy  scale  for 
the  item.  Under  such  circumstances  where  a task  is  performed  by  very 
few  graduates,  a recommendation  to  greatly  increase  training  emphasis 
would  be  suspect.  Other  factors  that  could  counter  Training  Adequacy 
recommendations  would  be  feedback  data  from  other  sources.  If  equipment 
breakdowns  are  numerous  for  a piece  of  equipment,  an  average  Training 
Adequacy  reconriendation  to  reduce  maintenance  training  for  the  item 
would  require  much  consideration  before  implementation.  Generally, 
however,  the  Fleet  personnel  are  as  much  aware  of  these  other  factors  as 
training  personnel,  and  in  most  instances,  these  average  ratings  of 
Training  Adequacy  can  be  heeded.  In  any  curriculum  modification,  the 
comments  and  recommendations  included  with  item  ratings  should  be  given 
much  consideration. 

Job  tasks  recommended  for  addition  to  the  curriculum  may  be  included 
if  enough  persons  (perhaps  10  percent  of  respondents)  recommend  them  and 
other  sources  agree  to  this  need.  A small  percentage  such  as  this  can 
be  acted  upon  since  many  other  persons  would  agree  to  the  need  but  did 
not  think  of  it  at  the  time  of  questionnaire  completion.  Other,  less 
verifiable,  recommendations  can  be  included  in  the  revised  questionnaire 
to  be  used  in  the  next  round  of  evaluations  to  determine  whether  or  not 
they  should  indeed  be  added  to  the  curriculum. 

Ideally,  the  revisions  should  be  made  and  the  new  curriculum 
implemented  within  six  months  following  the  analysis  of  questionnaire 
data.  Whatever  the  period  required  for  curriculum  revision,  the  field 
evaluation  process  can  be  repeated  beginning  six  months  after  the  first 
trainees  graduate  from  the  new  curriculum.  During  this  six  months 
between  graduation  and  questionnaire  mailing,  the  questionnaire  can  be 
brought  up-to-date  to  include  new  material  added  to  the  curriculum  and 
material  under  consideration  for  future  use.  As  classes  graduate,  the 
randomization  process  can  begin  to  select  persons  who  are  to  receive 
questionnaires  six  months  later.  If  possible,  persons  selected  to 
receive  trainee  forms  should  be  shown  the  questionnaire  and  made  aware 
that  they  will  be  expected  to  respond  to  the  inquiry. 
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After  six  months  of  questionnaire  mailings  and  when  it  is  determined 
that  no  more  questionnaires  are  forthcoming,  the  data  analysis  and 
curriculum  revision  process  can  begin  again.  This  cycle  of  six-months 
of  mail -out  interrogation--six  months  of  data  analysis  and  curriculum 
revision--duration  of  revised  course--six  months  of  feedback  questionnaire 
revision  and  respondent  selection--back  to  six-months  of  mail-out  interro- 
gation, will  take  18  months  plus  the  duration  of  the  training  course. 

For  most  Navy  courses,  the  time  period  would  be  slightly  less  than  two 
years  between  mail-out  interrogations.  With  today's  rapidly  changing 
Navy,  this  period  almost  guarantees  that  a new  set  of  training  problems 
will  be  ready  for  identification. 

In  addition  to  the  indications  of  need  for  training  change  resulting 
from  these  mail -out  instruments  and  procedures,  internal  sources  of 
training  problems  will  also  be  providing  input  to  the  curriculum  revision 
process.  Presumably  much  agreement  will  exist  between  the  recommendations 
from  both  sources.  Only  the  data  from  the  Fleet,  however,  can  establish 
for  certain  that  a training  objective  is  being  adequately  met,  and, 
perhaps  of  more  importance,  only  this  data  can  indicate  whether  existing 
training  objectives  are  appropriate  ones. 

SUMMARY  OF  RECOMMENDED  PROCEDURES 

A long  questionnaire,  based  on  specific  job  tasks,  is  recommended 
for  obtaining  feedback  data  in  all  Navy  schools. 

Ratings  should  be  obtained  for  each  specific  job  task  on  the 
frequency  of  the  task  and  the  adequacy  of  school  training  for  the  task. 

An  open-ended  question  should  be  included  to  get  at  missing 
training  objectives. 

. Questionnaires  should  be  mailed  to  both  trainees  and  supervisors. 

The  sample  to  be  tested  should  be  selected  randomly  from  all 
graduates  over  a six-month  period. 

Questionnaires  should  be  mailed  six  months  after  graduation. 

Enough  questionnaires  should  be  mailed  to  obtain  75  returned 
from  trainees  and  75  returned  from  supervisors. 

Questionnaires  for  supervisors  should  be  addressed  to  their 
Commanding  Officers. 

. Questionnaires  for  trainees  should  be  mailed  directly  to  the 
trainee. 
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Trainees  should  be  notified  that  they  will  receive  feedback 
questionnaires.  This  can  be  done  by  mail  or,  more  economically,  while 
they  are  still  in  school. 

. Follow-up  procedures  should  be  initiated  one  month  after 
mailing  of  the  questionnaires  in  order  to  increase  the  return  rate. 

Average  ratings  of  Frequency  and  Adequacy  of  School  Training 
for  Tasks  should  be  calculated  for  each  questionnaire  item. 

. At  least  the  top  10  percent  and  bottom  10  percent  of  the  tasks 
with  extreme  average  Adequacy  ratings  should  be  reviewed  and  a revision 
of  the  curriculum  made  where  necessary. 
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FIFTEEN  GENERAL  TASKS  OF  SHORT  QUESTIONNAIRE 
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FIFTEEN  GENERAL  TASKS  OF  SHORT  QUESTIONNAIRE 

1.  TYPE  MESSAGES  ON  A TELETYPEWRITER  KEYBOARD  USING  THE  TOUCH  TYPE 
METHOD. 

2.  PREPARE  TELETYPE  TAPES  OF  MESSAGES  WITH  ROUTING  INDICATORS  FOR 
TRANSMISSION  IN  AUTODIN  FORMAT  OR  MODIFIED  ACP  126  FORMAT. 

3.  PROCESS  OUTGOING  MESSAGES. 

4.  PROCESS  INCOMING  MESSAGES. 

5.  MAINTAIN  COMMUNICATION  MESSAGE  FILES. 

6.  OBSERVE  SAFETY  PRECAUTIONS  WHEN  WORKING  WITH  ELECTRONIC  EQUIPMENT. 

7.  SELECT,  SET  UP  AND  PATCH  TELETYPE  AND  VOICE  EQUIPMENT. 

8.  RESTORE  MALFUNCTIONING  SUB-SYSTEMS  TO  NORMAL  OPERATION. 

9.  PERFORM  PLANNED  MAINTENANCE  SUB-SYSTEMS  ACTIONS. 

10.  OPERATE  A FLEET  MULTICHANNEL  BROADCAST. 

11.  OPERATE  SHIP-TO-SHIP  AND  SHIP-TO-SHORE  TELETYPE  CIRCUITS. 

12.  OPERATE  THE  AN/FYA-71  (V)  DSTE  AUTODIN  TERMINAL. 

13.  OPERATE  THE  AN/FGC-73  (V)  MULTIPLE  ADDRESS  PROCESSING  UNIT. 

14.  EFFECT  DISTRESS  COMMUNICATIONS  USING  THE  RADIOTELEGRAPH  MODE  OF 
OPERATION. 

15.  MAINTAIN  SECURITY  OF  CLASSIFIED  MATERIAL  AND  COMMUNICATION. 
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APPENDIX  B 

134  SPECIFIC  TASKS  OF  LONG  QUESTIONNAIRE 
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134  SPECIFIC  TASKS  OF  LONG  QUESTIONNAIRE 

1.  TYPE  PRELIMINARY  CALLS  AND  OTHER  TRANSMISSIONS  ON  A TELETYPE  KEYBOARD, 
USING  THE  TOUCH  TYPE  METHOD. 

2.  CUT  TELETYPE  TAPES  OF  MESSAGES  DESTINED  FOR  TRANSMISSION  IN  EITHER 
AUTODIN  OR  MODIFIED  ACP-126  FORMAT. 

3.  CHECK  THE  ACCURACY  OF  PREPARED  TAPE. 

4.  READ  PRECUT  TELETYPE  TAPES  CONTAINING  NO  PRINTING  ON  THE  TAPE. 

5.  CORRECT  PRECUT  TELETYPE  TAPES. 

6.  CHECK  THE  TAPE  HEADING  BETWEEN  FORMAT  LINE  FOUR  AND  EOM. 

7.  PREPARE  TAPES  FROM  ORIGINATOR'S  ROUGH  DRAFTS. 

8.  PREPARE  HEADER  REQUIREMENTS. 

9.  IDENTIFY  COMMAND,  COLLECTIVE,  CONJUNCTIVE  AND  GEOGRAPHIC  ADDRESS 
GROUPS. 


10.  COMPLY  WITH  UNIT'S  OPERATIONAL  CHAIN  OF  COMMAND  WHEN  PROCESSING 
MESSAGES. 


11.  COMPLY  WITH  UNIT'S  ADMINISTRATIVE  CHAIN  OF  COMMAND  WHEN  PROCESSING 
MESSAGES. 

12.  CHECK  THE  VALIDITY  OF  THE  RELEASING  OFFICER'S  SIGNATURE  ON  EACH 
MESSAGE. 

13.  HANDLE  EACH  MESSAGE  IN  ACCORDANCE  WITH  THE  PRECEDENCE  ASSIGNED  BY 
THE  DRAFTER. 

14.  ASSIGN  TO  EACH  MESSAGE  A DATE-TIME  GROUP. 


15.  LOG  OUTGOING  MESSAGES  IN  THE  CENTRAL  MESSAGE  LOG. 


16.  DISTINGUISH  BETWEEN  VARIOUS  TELECOMMUNICATION  METHODS  OF  MESSAGE 
DELIVERY. 

17.  DETERMINE  THE  METHOD  OF  MESSAGE  DELIVERY  TO  BE  EMPLOYED  AND  THE 
FORMAT  REQUIRED. 
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18.  ENCODE/DECODE  ADDRESS  GROUPS. 

19.  ASSIGN  CALL  SIGNS,  ADDRESS  GROUPS,  & ROUTING  INDICATORS  (AS  REQUIRED). 

20.  IDENTIFY  INCORRECT  NAVAL  ACTIVITY  SHORT  TITLES  & CHANGE  THEM  TO  CONFORM 
TO  THE  PLAD. 

21.  IDENTIFY  ELEMENTS  OF  AUTODIN  HEADERS. 

22.  PLACE  MESSAGES  IN  AUTODIN  FORMAT. 

23.  CORRECT  FORMAT  ERRORS  IN  MESSAGES  FORMATTED  IAW  JANAP-128. 

24.  CONVERT  MESSAGES  IN  AUTODIN  FORMAT  TO  MODIFIED  ACP-126  FORMAT. 

25.  PLACE  RELAY  INSTRUCTIONS  ON  MESSAGE  HEADINGS. 

26.  CHECK  EACH  MESSAGE  FOR  PROPER  CLASSIFICATION  CRITERIA  INCLUDING 
SECURITY  WARNINGS  IN  FORMAT  LINES  2 & 4. 

27.  VERIFY  GEOGRAPHICAL  LOCATIONS  WHEN  PROCESSING  OUTGOING  MESSAGES. 

28.  IDENTIFY  CATEGORY  OF  PRECEDENCE. 

29.  COMPLY  WITH  HANDLING  TIME  OBJECTIVE  FOR  EACH  PRECEDENCE  CATEGORY. 

30.  VERIFY  EXISTENCE  OF  CLASSIFICATION  AND  DOWNGRADING/DECLASSIFICATION 
MARKINGS. 

31.  VERIFY  EXISTENCE  OF  STANDARD  SUBJECT  IDENTIFICATION  CODE  (SSIC). 

32.  SCREEN  MESSAGE  HEADINGS  FOR  MESSAGES  ADDRESSED  TO  ADDRESSEES  ON 
GUARDLIST. 

33.  RECORD  TIME  OF  RECEIPT  OF  EACH  MESSAGE  ADDRESSED  TO  GUARDLIST. 

34.  ALERT  PERSONNEL  WHEN  FLASH  MESSAGE  IS  RECEIVED  & PERFORM  THE 
PRESCRIBED  PROCESSING  ACTIONS  FOR  FLASH  TRAFFIC. 

35.  DETERMINE  THE  INCIDENCE  OF  RECEIPT  OF  DUPLICATES  OF  MESSAGES 
PREVIOUSLY  RECEIVED. 

36.  RECORD  THE  ORIGINATOR  AND  DATE-TIME  GROUP  OF  EACH  MESSAGE  RECEIVED. 

37.  USING  AN  INTERNAL  ROUTING  GUIDE,  INDICATE  THE  INTERNAL  ROUTING 
NECESSARY  FOR  EACH  MESSAGE  ADDRESSED  TO  GUARDLIST. 
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38.  MAKE  REPRODUCTION  OF  EACH  MESSAGE  ADDRESSED  TO  GUARDLIST  IN  SUFFICIENT 
QUANTITIES  TO  SATISFY  THE  INTERNAL  ROUTING  INDICATED. 


39.  MAKE  INTERNAL  DISTRIBUTION  OF  MESSAGES  RECEIVED  IN  ACCORDANCE  WITH 
THE  INTERNAL  ROUTING  INDICATED. 

40.  IDENTIFY  CALL  SIGNS. 


41.  IDENTIFY  BROADCAST  MESSAGES. 


42.  MONITOR  PAGE  PRINTERS  THAT  ARE  ELECTRICALLY  CONNECTED  INTO  THE  FLEET 
RADIOTELETYPE  BROADCAST  SUB-SYSTEM. 


43.  CHECK-OFF  BROADCAST  NUMBERS  BY  INDICATING  THE  CLASSIFICATION  OF  EACH 
MESSAGE  PASSED  ON  THE  BROADCAST. 

44.  DETERMINE  THE  INCIDENCE  OF  MISSING  BROADCAST  NUMBERS,  BY  CONTINUOUS 
NUMBER  CONTINUITY  CHECK. 

45.  RECEIVE  MESSAGES  VIA  THE  FLEET  BROADCAST. 


46.  MAINTAIN  BROADCAST  FILES. 


47.  FILE  ONE  COPY  OF  ALL  FIRST  RUN  TRAFFIC  IN  THE  BROADCAST  FILE  IN 
BROADCAST  NUMBER  ORDER. 

48.  DELIVER  ONE  COPY  OF  ALL  FIRST  RUN  TRAFFIC  TO  THE  BROADCAST  TRAFFIC 
CHECKER. 

49.  IDENTIFY  SPECIAL  CATEGORY  MESSAGES. 

50.  IDENTIFY  GENERAL  MESSAGES  & THEIR  SERIAL  NUMBERS. 

51.  IDENTIFY  READDRESSED  MESSAGES. 

52.  FILE  TOP  SECRET  AND  SPECIAL  CATEGORY  MESSAGES  IN  A CRYPTOCENTER  FILE 

IN  DATE-TIME  GROUP  ORDER,  AND  CONSTRUCT  A FILLER  FOR  EACH  MESSAGE  FILED. 

53.  FILE  GENERAL  MESSAGE  IN  A GENERAL  MESSAGE  FILE,  SEGREGATED  BY  GENERAL 
MESSAGE  TITLES  IN  SERIAL  NUMBER  ORDER:  CONSTRUCT  A FILLER  FOR  EACH 
MESSAGE  SO  FILED. 

54.  PREPARE  A FILLER  FOR  EACH  READDRESSAL  MESSAGE  THAT  IS  PROCESSED. 


55.  FILE  ALL  MESSAGES  (OTHER  THAN  TOP  SECRET,  SPECIAL  CATEGORY,  AND  GENERAL 
MESSAGES)  AND  FILLERS  IN  THE  COMMUNICATIONS  CENTER  FILE  IN  DATE-TIME 
GROUP  ORDER.  7/1 
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56.  CONDUCT  ROUTINE  DESTRUCTION  OF  CLASSIFIED  MATERIAL. 

57.  CONDUCT  EMERGENCY  DESTRUCTION  OF  CLASSIFIED  MATERIAL. 

58.  FILE  MONITOR  ROLLS,  RADIO  LOGS,  & SEND/RECEIVE  LOGS. 

59.  DETERMINE  THE  MEANING  OF  PROSIGNS. 

60.  DISTINGUISH  BETWEEN  PROSIGNS  & OPERATING  SIGNALS. 

61.  ENCODE/DECODE  VOICE  CALL  SIGNS. 

62.  DISTINGUISH  BETWEEN  SHIP/SHIP  & SHIP/SHORE  TELETYPE  CIRCUITS. 

63.  DISTINGUISH  BETWEEN  THE  THREE  MAJOR  COMPONENTS  OF  A NAVCOMMSTA  IN 
RELATION  TO  THEIR  CONTACT  WITH  THE  SHIPBOARD  SHIP/SHORE  OPERATOR. 

64.  ENCODE/DECODE  OPERATING  SIGNALS. 

65.  INITIATE  AND  ANSWER  PRELIMINARY  CALLS. 

66.  TRANSMIT  MESSAGES  IN  AUTODIN  AND  MODIFIED  ACP-126  FORMAT  IN  THE 
ORDER  OF  THE  PRECEDENCE  ASSIGNED. 

67.  PLACE  TAPES  IN  BACKLOG  BIN  BY  PRECEDENCE. 

68.  REQUEST,  AND  REPLY  TO  REQUESTS  FOR  REPETITIONS  AND  CORRECTIONS. 

69.  PROVIDE  RECEIPT  FOR  TRANSMISSIONS  AND  MESSAGES,  AFTER  ENSURING  THAT 
THEY  ARE  ERROR  FREE. 

70.  LOG  ALL  TRANSMITTED  AND  RECEIVED  MESSAGES  IN  THE  SEND  AND  RECEIVE 
LOGS,  RESPECTIVELY. 

71.  AFFIX  A TRANSMISSION  OR  RECEIVE  ENDORSEMENT  TO  EACH  MESSAGE 
TRANSMITTED  AND  RECEIVED,  RESPECTIVELY. 

72.  PERFORM  PAPER,  TAPE  AND  RIBBON  CHANGES,  AS  NECESSARY. 

73.  LOG  TIME  ENTRIES  IN  RADIOTELEGRAPH.  LOG  EACH  TRANSMISSION. 

74.  PERFORM  THE  RESCUE  OF  A PERSON  IN  CONTACT  WITH  A LIVE  CIRCUIT. 

75.  PERFORM  THE  FOLLOWING  FIRST  AID  PROCEDURES:  MOUTH-TO-MOUTH 
RESUSCITATION,  BACK-PRESSURE  ARM-LIFT  AND  BACK-PRESSURE  HIP-LIFT 
ARTIFICIAL  RESPIRATION,  TREATMENT  FOR  SHOCK,  AND  TREATMENT  FOR  BURNS. 
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76.  OBSERVE  SAFETY  PRECAUTIONS  WHEN  WORKING  WITH  ELECTRONIC  EQUIPMENT. 

77.  PERFORM  THE  NECESSARY  SAFETY  PROCEDURES  FOR  GOING  ALOFT. 

78.  IDENTIFY  NAVAL  COMMUNICATION  EQUIPMENT  BY  MEANS  OF  THE  JOINT 
ELECTRONICS  TYPE  DESIGNATOR  SYSTEM. 

79.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/WRC-1  TRANSCEIVER. 

80.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/URC-9  TRANSCEIVER. 

81.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/UCC-1  TELETYPE  TERMINAL 

EQUIPMENT. 

82.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/SGC-1A  TELETYPE  TERMINAL 
EQUIPMENT. 

83.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/SRC-20. 

84.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  R-1051  RECEIVER. 

85.  ADJUST  THE  FRONT  PANEL  CONTROLS  OF  THE  AN/URT-23  TRANSMITTER. 

86.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/URA-17  TELETYPE  CONVERTER. 

87.  ADJUST  THE  OPERATING  CONTROLS  OF  THE  AN/WRT-2  TRANSMITTER. 

88.  ACTIVATE  UNCOVERED  MF/HF  VOICE  SUB-SYSTEM  (TYPE  Y). 

89.  ACTIVATE  UNCOVERED  UHF  VOICE  SUB-SYSTEM  (TYPE  U). 

90.  ACTIVATE  COVERED  UHF/VHF  VOICE  SUB-SYSTEM  (TYPE  R). 

91.  ACTIVATE  UHF  ORESTES  COVERED  SIMPLEX  SUB-SYSTEM  (TYPE  B). 

92.  ACTIVATE  UHF  ORESTES  COVERED  DUPLEX  SUB-SYSTEM  (TYPE  C). 

93.  ACTIVATE  MF/HF  ORESTES  COVERED  SIMPLEX  SUB-SYSTEM  (TYPE  D). 

94.  ACTIVATE  MF/HF  ORESTES  COVERED  DUPLEX  SUB-SYSTEM  (TYPE  G). 

95.  ACTIVATE  ORESTES  COVERED  SHIP-SHORE  MULTIPLEX  TERMINATION  (TYPE  P). 

96.  ACTIVATE  COVERED  FLEET  MULTICHANNEL  BROADCAST,  RECEIVER  (TYPE  N). 
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97.  DETERMINE  THE  EFFECTS  OF  THE  IONOSPHERE  ON  SKIP  DISTANCES  AND  SKIP 
ZONE,  AND  USE  THE  RESULTS  OF  THIS  DETERMINATION  AS  AN  AID  IN 
ESTABLISHING  A LONG  DISTANCE  COMMUNICATION  PATH. 

98.  DETERMINE  THE  EFFECTS  OF  THE  IONOSPHERE  ON  SKYWAVE  PROPAGATION  AND 
USE  THE  RESULTS  OF  THIS  DETERMINATION  AS  AN  AID  IN  ESTABLISHING  A 
LONG  DISTANCE  COMMUNICATION  PATH. 

99.  TUNE  ANTENNAS  TO  ACHIEVE  OPTIMUM  TRANSMISSION. 

100.  OPERATE  THE  KWX-8  CONTROL  UNIT  ON  CRYPTO  EQUIPMENT. 

101.  CHECK  SUB-SYSTEM  OPERATION  FOR  INDICATIONS  OF  SUB-NORMAL  PERFORMANCE. 

102.  ISOLATE  THE  INCIDENCE  OF  SUB-NORMAL  PERFORMANCE  TO  IMPROPERLY 
ADJUSTED  EQUIPMENT,  MALFUNCTIONING  EQUIPMENT,  FAULTY  PATCHES,  OR 
FAULTY  COMMUNICATION  PATH. 

103.  RESTORE  SYSTEMS  TO  NORMAL  OPERATION  BY  CORRECTING  THE  PATCH, 
COMMUNICATION  PATH  SUBSTITUTION,  OR  BY  READJUSTMENT  OR  REPLACEMENT 
OF  IMPROPERLY  ADJUSTED  OR  FAULTY  EQUIPMENT. 

104.  LOCATE  PMS  ASSIGNMENT  ON  THE  3-M  WEEKLY  SCHEDULE. 

105.  LOCATE  THE  MAINTENANCE  REQUIREMENT  CARD  (MRC)  WHICH  PERTAINS  TO 
THE  PMS  ACTION  ASSIGNED. 

106.  LOCATE  THE  SPECIFIC  EQUIPMENT  ON  WHICH  THE  PMS  ACTION  IS  TO  BE 
PERFORMED. 

107.  PERFORM  THE  PMS  ACTION  CALLED  FOR  BY  THE  MRC. 

108.  PAINT  ANTENNAS. 

109.  CLEAN  ANTENNAS. 

110.  PERFORM  RESISTANCE  CHECKS  ON  ANTENNAS. 

111.  PERFORM  VISUAL  INSPECTION  OF  ANTENNAS. 

112.  OPERATE  PAPER  TAPE  READER,  CARD  READER  & SEND  PORTION  OF  THE 
COMMON  CONTROL  UNIT  OF  THE  AN/FYA-71 (V)  DSTE  AUTODIN  TERMINAL. 

113.  CONDUCT  CONTINUITY  CHECK  OF  THE  MULTIPLE  ADDRESS  PROCESSING  UNIT 
SYSTEM  EVERY  HOUR  ON  THE  HALF  HOUR. 
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114.  RESPOND  IN  THE  PRESCRIBED  MANNER  TO  ANY  ALARM  CONDITION  IN  THE 
MULTIPLE  ADDRESS  PROCESSING  UNIT  SYSTEM. 

115.  OPERATE  THE  TT-329  (REPERFORATOR). 

116.  OPERATE  THE  TT-331  (TWO  ROWS  OF  THREE  REPERFORATORS). 

117.  MAINTAIN  BROADCAST  STATUS  LOG. 

118.  INITIATE/RESPOND  TO  PRELIMINARY  CALLS  AS  THE  NET  CONTROL  STATION. 

119.  TRANSMIT  CARD  TRAFFIC. 

120.  CLOSE  OUT  SEND  CHANNELS. 

121.  DELIVER  ALL  MESSAGE  TAPES  RECEIVED  TO  THE  MULTIPLE  ADDRESS 
PROCESSING  UNIT  SYSTEM  OPERATOR. 

122.  USE  THE  INTERNATIONAL  DISTRESS  AND  CALLING  FREQUENCY  AND  OBSERVE 
THE  SILENT  PERIODS. 

123.  RESPOND  TO  DISTRESS,  URGENCY,  AND  SAFETY  SIGNALS,  AND  TO  THE  FIVE 
ENEMY  CONTACT  ALARM  SIGNALS  EMPLOYED  BY  MERCHANT  VESSELS  IN  TIME 
OF  WAR. 

124.  PROVIDE  RECEIPT  FOR,  OR  RELAY,  IN  INTERNATIONAL  FORM,  INTERCEPTED 
DISTRESS  MESSAGES  AND  ENEMY  CONTACT  REPORTS,  USED  BY  MERCHANT 
VESSELS  IN  TIME  OF  WAR. 

125.  PREPARE  A MESSAGE  FOR  TRANSMISSION  BY  RADIOTELEGRAPH  IN  PLAINDRESS, 
ABBREVIATED  PLAINDRESS,  AND  CODRESS  FORM. 

126.  OPERATE  A RADIOTELEGRAPH  CIRCUIT  USING  MILITARY  PROCEDURES  AND  EMPLOY- 
ING CALL  SIGN  ENCRYPTION  AND  AUTHENTICATION. 

127.  USE  THE  COUNTERMEASURES  AVAILABLE  TO  REDUCE  THE  EFFECTS  OF  JAMMING  AND 
HARMFUL  INTERFERENCE. 

128.  RECORD  THE  REQUIRED  INFORMATION  NECESSARY  FOR  REPORTS  WHEN  CONDITIONS 
OF  JAMMING  OR  HARMFUL  INTERFERENCE  ARE  DETECTED. 

129.  ENCODE/DECODE  INTERNATIONAL  CASS  SIGNS. 

130.  ENCODE/DECODE  TASK  ORGANIZATION  CALL  SIGNS. 
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131.  GUARD  AGAINST  ANY  INCIDENCE  OF  COMPROMISE  OF  CLASSIFIED  MATERIAL 
OR  ANY  SECURITY  VIOLATION. 

132.  EMPLOY  PHYSICAL  SECURITY  MEASURES  BY  ADHERENCE  TO  THE  ACCOUNTING, 
DISSEMINATION,  & STOWAGE  PROCEDURES  PRESCRIBED  FOR  CLASSIFIED 
MATERIAL. 

133.  EMPLOY  TRANSMISSION  SECURITY  MEASURES  BY  ADHERENCE  TO  PRESCRIBED 
TRANSMISSION  PROCEDURES  AND  BY  ALERTNESS  TO  AND  REPORTING  OF 
DEVIATIONS  FROM  THESE  PROCEDURES. 

134.  PROJECT  CRYPTOSECURITY  BY  THE  PROPER  USE  OF  CRYPTOGRAPHIC  MATERIAL, 
CRYPTO  SYSTEMS,  AND  RELATED  CRYPTO  MATERIAL. 
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APPENDIX  C 

BASIC  RATING  SCALE  FORM  USED  ON  LONG 
AND  SHORT  QUESTIONNAIRES 
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Listed  below  is  a duty  and  the  tasks  which  it  includes.  Rate  each  task 
on  the  three  scales  by  circling  the  appropriate  number.  If  you  have 
any  specific  comments  about  training  for  these  tasks,  please  write 
them  in  the  area  below  the  question  or.  if  more  space  is  needed,  on  the 
back  of  this  sheet. 

FREQUENCY  OF  TASK 

CRITICALITY  OF  TASK 

PERFORMANCE  OF  TASK 
UPON  ARRIVAL  FROM  SCHOOL 

woe:  —EIGHTEEN  N 

UUTY;  PERFORM  THE  FOLLOWING  SHORE  STATION 
COMMUNICATION  CENTER  FUNCTIONS 

1.  Never  performed. 

2.  Performed  very  infrequently. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  dally. 

1.  Very  small  Importance 

2.  Small  Importance. 

3.  Moderate  Importance 

4.  High  importance 

5.  Very  high  Importance. 

L Could  not  perform. 

2.  Substantially  Inadequate 

3.  Somewhat  inadequate. 

4.  Adequate 

5.  More  than  adequate. 

112.  OPERATE  PAPER  TAPE  READER,  CARD  READER 

& SEND  PORTION  OF  THE  COMMON  CONTROL  UNIT 
OF  THE  AN/FYA-71 (V)  DSTE  AUTODIN  TERMINAL. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

113.  CONDUCT  CONTINUITY  CHECK  OF  THE  MULTIPLE 
ADDRESS  PROCESSING  UNIT  SYSTEM  EVERY  HOUR 
ON  THE  HALF  HOUR. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

TT4T  RESPOND” IN  THE  PRESCRIBED  MANNER  TO 
cx.  ANY  ALARM  CONDITION  IN  THE  MULTIPLE 
™ ADDRESS  PROCESSING  UNIT  SYSTEM. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

115.  OPERATE  THE  TT-329  (REPERFORATOR). 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

116.  OPERATE  THE  TT-331  (TWO  ROWS  OF  THREE 
REPERFORATORS) . 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

117.  MAINTAIN  BROADCAST  STATUS  LOG. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

118.  INITIATE/RESPOND  TO  PRELIMINARY  CALLS  AS 
THE  NET  CONTROL  STATION. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

119.  TRANSMIT  CARD  TRAFFIC. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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ANSWER  THE  FOLLOWING  QUESTIONS  BY  CIRCLING  "YES"  OR  "NO" 


1.  Is  there  any  part  of  your  job  that  you  cannot  do  because  you  were  not  taught  how  to 
do  it  in  school? 

Yes 

No 

2.  Is  there  anything  you  have  had  to  learn  to  do  on  the  job  that  was  not  taught  in 
school  but  should  have  been? 

Yes 

No 

3.  Is  there  anything  that  you  received  some  training  on  in  school,  but  not  enough? 

Yes 

Ho 

4.  Did  the  school  actually  teach  the  wrong  way  to  do  any  part  or  parts  of  your  job? 

Yes 

No 

5.  Did  you  learn  anything  in  school  that  does  not  help  you  in  your  job  in  any  way? 

Yes 

No 


6. 


Is  there  something  that  you  do  well  on  your  job  that  does  not  require  as  much  time  or 
emphasis  as  it  received  in  school? 

Yes 

No 


7.  Did  the  school  teach  you  to  do  anything  that  you  could  have  learned  to  do  better  or 
faster  on  the  job? 

Yes 

No 


8.  Please  return  to  those  Questions  1 through  7 that  are  answered  "Yes”  and,  if  possible, 
give  one  or  more  specific  examples  to  help  the  school  identify  and  correct  its  problems. 
Please  do  this  carefully  and  thoughtfully.  You,  as  a graduate  of  the  school  working  in 
the  job  you  were  trained  for,  are  in  an  ideal  position  to  identify  the  problems  that  are 
co  almost  certain  to  exist  in  the  school. 
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CHIEF  OF  NAVAL  EDUCATION  AND  TRAINING 
NAVAL  AIR  STATION 
PENSACOLA,  FLORIDA  32908 


Code  N-343 
1 0 JUL  1974 


CO 

CO 


From:  Chief  of  Naval  Education  and  Training 

To:  Recent  Graduates  of  Radioman  "A"  School 

Subj : Field  Evaluation  of  Radioman  "A"  School  Training 

Enel:  (1)  Field  Evaluation  Materials 

1.  In  order  to  improve  performance  of  the  school  graduate  in  his 
assigned  unit.  Information  is  required  as  to  the  adequacy  and  relevancy 
of  your  training. 

2.  As  a recent  graduate  of  the  Radioman  "A"  School,  you  are  in  a 
unique  position  to  provide  information  to  the  schools  on  whether  or 
not  the  Radioman  "A"  School  has  adequately  prepared  you  to  perform  your 
job.  The  enclosed  forms  were  designed  to  aid  you  in  this  task. 

3.  Please  follow  instructions  carefully  and  provide  all  of  the 
Information  requested  on  these  forms.  Your  assistance  is  appreciated 
and  will  surely  enhance  the  efforts  to  improve  the  training  of  future 
Radiomen. 
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INSTRUCTIONS  TO  TRAINEE  FOR  COMPLETING  THE  RADIOMAN  "A"  SCHOOL  FEEDBACK  QUESTIONNAIRE 


Tasks  included  in  your  training  as  a Radioman  are  listed  in  this  booklet.  Rate 
each  particular  task  on  each  of  the  three  scales  to  the  right  of  the  task.  The 
"Frequency  of  Task"  scale  includes  levels  of  (1)  Never  Performed,  (2)  Performed  Very 
Infrequently,  (3)  Performed  Monthly,  (4)  Performed  Weekly,  and  (5)  Performed  Daily. 

Circle  the  number  corresponding  most  closely  to  the  frequency  with  which  you  perform 
the  task.  The  second  scale  is  "Criticality  of  the  Task,"  and  the  levels  are  (1)  Very 
Small  Importance,  (2)  Small  Importance,  (3)  Moderate  Importance,  (4)  High  Importance, 
and  (5)  Very  High  Importance.  Knowing  how  to  perform  a task  that  you  presently  don't 

V£> 

° do,  may  still  aid  you  in  your  job,  so  a task  that  you  "Never  Perform"  may  still  have 
some  importance.  The  final  scale  is  "Performance  of  Task  Upon  Arrival  from  School." 
Levels  are  (1)  Could  Not  Performj(2)  Substantially  Inadequate,  (3)  Somewhat  Inadequate, 

(4)  Adequate,  and  (5)  More  than  Adequate.  If  mess-cooking  or  other  activities  at  your 
duty  station  delayed  engagement  in  these  tasks,  rate  your  first  performance  of  the  task. 
If  you  never  perform  the  task  in  your  present  assignment,  estimate  how  well  your  training 
has  prepared  you  to  perform  it. 

When  rating  the  "importance"  of  a task,  you  are  expected  to  use  your  own  judgment. 
There  are  no  absolute  rules  to  guide  you  and  it  is  quite  likely  that  other  trainees  or 
supervisors  would  rate  tasks  differently  than  you.  The  same  considerations  apply  when 
rating  your  performance  on  a task  upon  arrival  from  school.  Your  evaluation  of  how 


well  you  are  able  to  perform  a task  may  be  different  than  an  evaluation  made  by  your 
supervisor. 

Please  remember,  our  goal  is  to  find  out  how  well  the  Radioman  School  is  preparing 
trainees  to  do  their  job.  It  is  not  to  rate  you  as  a trainee.  In  other  words,  if 
there  are  important  tasks  that  you  feel  you  could  not  adequately  perform  upon  arrival 
from  school,  then  it  is  apparent  there  is  a shortcoming  in  training.  We  are  interested 
in  your  independent  ratings  and  request  that  you  work  alone  when  answering  this  question- 
naire. Please  be  objective  and  accurate  and  return  the  completed  questionnaire  to  us  in 
the  enclosed  addressed  envelope.  All  replies  will  be  held  in  strictest  confidence  by  the 
study  team.  Thank  you  for  your  help! 

NOTE:  Please  be  sure  that  you  always  check  one  of  the  levels  in  the  "Frequency  of  Task" 

and  "Performance  of  Task  Upon  Arrival"  scales.  Even  though  you  might  never  have 
performed  some  of  the  tasks,  you  should  be  able  to  make  an  estimate  about  how  well 
you  could  perform  them  if  you  were  required  to. 
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PLEASE  FURNISH  THE  FOLLOWING  INFORMATION 


1. 

Name 

2. 

Rank 

3.  SSN 

4. 

Today's  date 

5. 

Present 

Duty 

Station 

6. 

How  many  months 

have  you  been  at  your 

present 

duty 

station? 

Date  arrived 7.  Have  you  been  assigned  to  duty  within  your  job 

specialty? If  No,  what  is  your  assignment? 

8.  Did  Mess-Cooking  or  other  activities  at  this  duty  station  delay  the  beginning  of 

work  in  your  specialty? If  Yes,  what  activities? 

How  long  was  the  delay? 

us  9.  Did  you  attend  International  Morse  Code  School  or  any  other  school  after  Radioman 

"A"  School?  If  Yes,  which  school? 

10.  Check  the  following  statement  that  best  describes  how  much  the  training  you  received 

at  the  Radioman  "A"  School  is  used  in  your  present  job. 

VERY  MUCH  MUCH  SOME 


VERY  LITTLE 


NOT  AT  ALL 
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Although  we  have  asked  you  to  consider  all  of  the  Radioman  training  in  "excruciating"  detail, 
there  is  still  much  that  you  can  do  for  us.  We  need  to  know  things  that  Radiomen  need  to  know  on 
their  jobs  that  presently  are  not  taught  in  "A"  School  and  which  should  be  taught  there.  Please 
indicate  any  such  areas  that  you  feel  should  be  included  in  "A"  School  training  in  the  future. 
Every  suggestion  you  make  will  be  submitted  to  curriculum  design  personnel  for  evaluation  and 


consideration. 


TAEG  Report  No.  19 


APPENDIX  I 

CARD  SORT  "PLACEMAT"  FOR  TRAINEE 


97 


TASK  IS  NEVER  PERFORMED  AT  THIS  UNIT 

. COULD  NOT  PERFORM  THIS  TASK 

SUBSTANTIALLY  INADEQUATE  PERFORMANCE 

. * 

SOMEWHAT  INADEQUATE  PERFORMANCE 

ADEQUATE  PERFORMANCE 

MORE  THAN  ADEQUATE  PERFORMANCE 

ENCLOSED  IS  A SET  OF  CAROS  WITH  THE  TASKS  INCLUDED  IN  THE  TRAINING  RECEIVED  BY  THE  RADIOMAN  YOU  SUPERVISE.  SORT  EACH  CARD  INTO  THE  APPROPRIATE 
BOX  DEPENDING  UPON  HOW  WELL  THE  TASK  WAS  PERFORMED  UPON  ARRIVAL  FROM  SCHOOL.  IF  PERFORMANCE  ON  A TASK  WAS  SUBSTANTIALLY  INADEQUATE.  THE  III  IS 
VERY  LIKELY  A SHORT-COMING  IN  TRAINING  FOR  THAT  TASK.  IF  YOU  HAVE  ANY  SUGGESTIONS  TO  IMPROVE  OR  REDUCE  THE  COST  OF  TRAINING  FOR  AS^  lAih.  PLEASE 
WRITE  IT  ON  THE  TASK  CARD  ITSELP.  WHEN  FINISHED,  PLACE  THE  APPROPRIATE  LABEL  AROUND  EACH  STACK  AND  RETURN  THE  MATERIALS  IN  THE  i£I.F- ADDRESS  ED  ENVELOPE 
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SAMPLE  LETTER  TO  SUPERVISOR 


From:  Curriculum  Update  Division  School 

To:  Supervisor  of  Seaman  

Subj : Field  Evaluation  of  School  training 

Enel:  (1)  Field  Evaluation  Materials 

1.  As  an  experienced  person  in  your  rating  and  a supervisor 
of  a recent  school  graduate,  you  are  in  an  ideal  position  to 
tell  us  whether  our  graduates  are  meeting  job  requirements 

at  your  unit.  The  enclosed  materials  make  it  possible  for  you 
to  indicate  whether  too  much  or  too  little  emphasis  was  given 
to  any  of  the  various  tasks  covered  in  school.  On  the  final 
page  of  this  questionnaire  we  request  that  you  indicate  job 
tasks  that  are  not  presently  covered  in  school,  but  which 
should  be  covered  in  the  future.  Throughout  your  completion 
of  these  materials,  we  hope  you  will  write  down  any  thoughts 
you  may  have  about  training  problems,  recommendations  for 
their  solution,  and  any  other  aspects  of  school  training. 

2.  Please  return  these  materials  in  the  enclosed  envelope 
within  two  weeks,  if  possible.  This  information  will  aid  us 

to  provide  better  training  of  

in  the  future. 

3.  If  you  have  recently  completed  field  evaluation  materials 

for School  there  is  no  need  to  complete 

these  unless  you  have  some  additional  recommendations.  However, 
we  would  appreciate  if  you  would  pass  these  materials  on  to 

some  other  experienced  who  is 

familiar  with  the  above  person's  work. 


CHIEF  PETTY  OFFICER 


PLEASE  FURNISH  THE  FOLLOWING  INFORMATION  ABOUT  YOURSELF  AND  THE  RECENT  RADIOMAN  "A" 
SCHOOL  GRADUATE  WHOM  YOU  SUPERVISE 


1. 

Your  name 

2. 

Rank 

3 . Today ' s 

Date 

4. 

Duty  Station 

5. 

Name 

of  recent  RM  "A"  School 

graduate 

6.  His  rank  7.  How  many  months  has  he  been 

assigned  to  his  present  duty  station? 8.  Has  he  been  assigned  to  duty  within 

his  job  specialty? If  No,  what  is  his  assignment? 

9.  Did  he  attend  International  Morse  Code  School  or  any  other  school  after  completing 

Radioman  "A"  School? If  Yes,  which  school? 

10.  Did  mess-cooking  or  other  activities  at  this  duty  station  delay  the  beginning  of  his 

work  in  his  specialty? If  Yes,  what  activities? 

How  long  was  the  delay? 11.  Check  the  following  statement  that  best 

describes  how  much  this  man's  RM  "A”  School  training  is  utilized  in  his  present  job? 


VERY  MUCH 


MUCH 


SOME 


VERY  LITTLE 


NOT  AT  ALL 
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INSTRUCTIONS  TO  SUPERVISOR  FOR  COMPLETING  RATING  SCALES 


On  the  following  pages  tasks  are  listed  which  received  at  least  some  emphasis  in 
school.  Please  rate  each  task  on  the  two  scales  at  the  right  of  the  task  by  circling 
the  most  appropriate  number.  On  the  "Frequency  of  Task"  scale,  select  the  category 
that  corresponds  most  closely  to  the  actual  frequency  with  which  this  task  is  performed 
by  the  recent  school  graduate  that  you  supervise  in  his  present  assignment. 

On  the  "Adequacy  of  School  Training  for  This  Task"  scale,  select  the  most  appro- 
priate of  the  following  categories: 

1.  Task  requires  much  more  emphasis  in  school. 

2.  Training  less  than  adequate  for  task,  increase  emphasis. 

3.  Training  adequate  for  task. 

4.  Training  more  than  adequate  for  task,  reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for  this  task. 

(You  may  skip  this  "Adequacy"  rating  for  a particular  task  if  that  task  is  never 
performed  and  you  do  not  feel  you  can  rate  adequacy  of  training  for  it. ) 

In  making  this  rating  consider  such  things  as  the  following: 

Problems  he  may  have  had  performing  this  task  when  first  required  to  do  it; 
The  amount  of  time  that  was  required  by  you  or  by  others  at  your  unit  to 
bring  him  "up-to-speed" ; 


Whether,  for  some  reason,  the  task  should  have  been  learned  on-the-job 


instead  of  in  school;  and 

Whether  learning  to  perform  this  task  in  school  does  not  help  this  man  in 
his  present  job  or  will  not  help  him  in  the  foreseeable  future. 

Your  experience  in  your  rating  makes  you  uniquely  qualified  to  judge  when  job 
tasks  need  more  or  less  school  emphasis.  Not  only  have  you  already  considered  the 
question  of  what  is  the  proper  balance  between  school  training  and  training  on  the 
job,  but  you  can  also  see  the  possible  future  value  of  training  that  has  little 
immediate  use.  We  look  forward  to  seeing  your  ratings  of  training  adequacy  and  will 
give  them  much  consideration. 


r 


PAGE: 


FREQUENCY  OF  TASK 


Listed  below  are  tasks  which  presently  receive  at  least  some  emphasis  in  school. 
Please  rate  each  task  on  the  scales  at  the  right  by  circling  the  most  appropriate 
number.  Please  feel  free  to  also  include  your  reasons  for  your  rating  and/or  any 
specific  recommendations  for  training  on  this  task.  Your  comments  may  be  written 
in  any  available  space  on  the  front  or  back  of  this  page  or  on  a separate  sheet. 


L Never  performed.* 

2.  Seldom  performed  or  only  in  emergencies. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  daily. 


L 

2. 

3. 

4. 


5. 


•ADEQUACY  scale  may  be  skipped  if  task  is 
never  performed. 


1 2 3 4 5 


1 2 3 4 5 


ADEQUACY  OF  SCHOOL  TRAINING 
FOR  THIS  TASK 


Task  requires  much  more  emphasis  In  school. 
Training  less  than  adequate  for  task. 

Increase  emphasis. 

Training  adequate  for  task. 

Training  more  than  adequate  for  task, 
reduce  emphasis. 

Greatly  reduce  or  eliminate  training  for 
this  task. 


1 2 3 4 5 


1 2 3 4 5 


O 

4^ 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


2 


2 


2 


2 


2 


2 


3 4 


3 4 


3 4 


3 4 


3 4 


3 4 


5 


5 


5 


5 


5 


5 


Although  we  have  already  asked  you  to  consider  existing  school  training  in  great 
detail,  there  is  one  more  very  important  job  you  can  do  for  us.  We  need  to  know  what 
things  presently  are  NOT  taught  in  school  but  should  be  taught  there.  Consider  things 
the  trainee  has  had  to  learn  on  the  job  with  much  loss  of  time  for  both  him  and  his 
supervisors.  Also  consider  tasks  he  still  cannot  perform  because  he  did  not  learn  them 
in  school  and  because  it  has  not  been  possible  to  train  him  on  the  job.  Please  do  this 
carefully  and  thoughtfully.  As  a supervisor  of  a recent  school  graduate,  you  are  in  a 
unique  position  to  identify  those  things  which  are  almost  certain  to  be  missing  from 
school . 

O 

^ 1. 


2. 


3. 


4. 
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SAMPLE  LETTER  TO  TRAINEE 


From: 

Curriculum  Update  Division 

School 

To: 

Seaman 

Subj : 

Field  Evaluation  of 

School  Training 

Enel: 

(1)  Field  Evaluation  Materials 

1.  At  one  of  your  last  classes  in  school  the  important  task  you  can 
perform  in  the  identification  of  training  problems  was  discussed.  At 
this  time,  we  are  asking  you  to  aid  us  in  this  task  since  you  have 
probably  been  on  the  job  long  enough  to  have  developed  a good  under- 
standing of  your  duties  and  the  training  needed  to  perform  them. 

2.  The  enclosed  materials  make  it  possible  for  you  to  indicate 
whether  too  much  or  too  little  emphasis  was  given  to  any  of  the 
various  tasks  covered  in  school.  On  the  final  page  of  this  questionnaire 
we  request  that  you  indicate  job  tasks  that  are  not  presently  covered  in 
school,  but  which  should  be  covered  in  the  future.  Throughout  your 
completion  of  these  materials  we  hope  you  will  write  down  any  thoughts 
you  may  have  about  training  problems,  recommendations  for  their  solution, 
and  any  other  aspects  of  school  training. 

3.  Please  return  these  materials  in  the  enclosed  envelope  within  two 

weeks,  if  possible.  This  information  will  aid  us  to  provide  better 
training  of  in  the  future. 


Chief  Petty  Officer 


PLEASE  FURNISH  THE  FOLLOWING  INFORMATION 


1. 

Name 

2. 

Rank 

3.  SSN 

4. 

Today's  date 

5. 

Present 

Duty  Station 

6. 

How  many  months 

have  you  been  at  your 

present 

duty  station? 

Date  arrived 7.  Have  you  been  assigned  to  duty  within  your  job 

specialty? If  No,  what  is  your  assignment? 

8.  Did  Mess-Cooking  or  other  activities  at  this  duty  station  delay  the  beginning  of 

work  in  your  specialty? _ If  Yes,  what  activities? 

How  long  was  the  delay? 

9.  Did  you  attend  International  Morse  Code  School  or  any  other  school  after  Radioman 

"A"  School? If  Yes,  which  school?^ 

10.  Check  the  following  statement  that  best  describes  how  much  the  training  you  received 
at  the  Radioman  "A"  School  is  used  in  your  present  job. 

VERY  MUCH  MUCH  SOME 


VERY  LITTLE 


NOT  AT  ALL 


INSTRUCTIONS  TO  TRAINEE  FOR  COMPLETING  RATING  SCALES 


On  the  following  pages  tasks  are  listed  which  received  at  least  some  emphasis  in 
school.  Please  rate  each  task  on  the  two  scales  at  the  right  of  the  task  by  circling 
the  most  appropriate  number.  On  the  "Frequency  of  Task"  scale,  select  the  category 
that  corresponds  most  closely  to  the  actual  frequency  with  which  this  task  is  performed 
by  you  in  your  present  assignment. 

On  the  "Adequacy  of  School  Training  for  This  Task"  scale,  select  the  most  appro- 
priate of  the  following  categories: 

1.  Task  requires  much  more  emphasis  in  school. 

2.  Training  less  than  adequate  for  task,  increase  emphasis. 

3.  Training  adequate  for  task. 

4.  Training  more  than  adequate  for  task,  reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for  this  task. 

(You  may  skip  this  "Adequacy"  rating  for  a particular  task  if  that  task  is  never 
performed  and  you  do  not  feel  you  can  rate  adequacy  of  training  for  it.) 

In  making  this  rating  consider  such  things  as  the  following: 

Problems  you  may  have  had  performing  this  task  when  first  required  to  do  it; 
The  amount  of  time  that  was  required  by  your  supervisor  or  others  at  your 
unit  to  bring  you  "up- to-speed"  on  the  task; 


’ Whether,  for  some  reason,  the  task  should  have  been  learned  on-the-job 


instead  of  in  school;  and 

Whether  learning  to  perform  this  task  in  school  does  not  help  you  in  your 
present  job  or  will  not  help  you  in  the  foreseeable  future. 

Also  consider  that  school  training  is  expensive  and  must  be  used  only  for 
essential  tasks.  On  the  other  hand,  remember  that  operational  units  have  many  other 
functions  to  perform  beside  on-the-job  training. 

As  you  can  see,  the  rating  of  training  adequacy  is  not  simple.  We  are  asking 
you  to  do  this  since  you  hold  two  views  of  the  world  that  are  critical  for  judging  the 
adequacy  of  training.  One  view  is  of  school  training  as  it  exists  for  the  student  and 
the  other  view  is  of  the  requirements  of  your  present  job.  These  unique  perspectives 
of  yours  make  your  careful  ratings  invaluable  to  us! 


— J 

no 

PAGE 

FREQUENCY  OF  TASK 

ADEQUACY  OF  SCHOOL  TRAINING 
FOR  THIS  TASK 

Listed  below  are  tasks  which  presently  receive  at  least  some  emphasis  In  school. 
Please  rate  each  task  on  the  scales  at  the  right  by  circling  the  most  appropriate 
number.  Please  feel  free  to  also  include  vour  reasons  for  your  rating  and/or  any 
specific  recommendations  for  training  on  this  task.  Your  comments  may  be  written 

L Ne\ cr  performed.* 

2.  Seldom  performed  or  only  In  emergencies. 
X Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  daily. 

•ADEQUACY  scale  may  be  skipped  if  task  Is 
never  performed. 

L Task  requires  much  more  emphasis  In  school. 

2.  Training  less  tkarT adequate  for  task. 

Increase  emphasis. 

3.  T raining  adequate  for  task. 

4.  Training  more  than  adequate  for  task, 
reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for 
this  task. 

in  any  available  space  on  the  front  or  back  of  this  page  or  on  a separate  sheet. 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

Although  we  have  already  asked  you  to  consider  existing  school  training  in  great 
detail,  there  is  one  more  very  important  job  you  can  do  for  us.  We  need  to  know  what 
things  presently  are  NOT  taught  in  school  but  should  be  taught  there.  Consider  things 
you  have  had  to  learn  on  the  job  with  much  loss  of  time  for  both  you  and  your  super- 
visors. Also  consider  tasks  you  still  cannot  perform  because  you  did  not  learn  them  in 
school  and  because  it  has  not  been  possible  to  train  you  on  the  job.  Please  do  this 
carefully  and  thoughtfully.  As  a school  graduate  working  in  the  job  you  were  trained 
to  do,  you  are  in  a unique  position  to  identify  those  things  which  are  almost  certain 
to  be  missing  from  school. 

1. 


2. 


3. 


4. 


- 
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APPENDIX  L 


RECOMMENDED  REMINDER  LETTER 


From:  Curriculum  Update  Branch,  School 

To:  Recent  School  Graduate 


About  one  month  ago  you  were  mailed  materials  for  evaluating  the 

training  at School.  It  is  most  important  that  we  receive 

your  completed  questionnaire  in  order  that  we  may  use  the  data  in  our 

continuing  program  to  make  training  of both  relevant  and 

effective. 

If  you  did  not  receive  the  materials,  please  contact  us  and  we 
will  mail  another  set.  If  you  did  receive  them,  please  complete  and 
return  them  as  soon  as  possible. 

Sincerely, 

Chief  Petty  Officer  
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ITEM  # 52 


Use  principles  of  sky  wave  propagation  to  establish  a long  distance 
communication  path. 


TRAINEE 

FREQUENCY  OF  TASK 


L Never  performed.* 

2.  Seldom  performed  or  only  In  emergencies. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  dally. 


•ADEQUACY  scale  may  be  skipped  If  task  Is 
never  performed. 


1 2 3 4 5 


TRAINEE 


SUPERVISOR 


ADEQUACY  OF  SCHOOL  TRAINING 
FOR  THIS  TASK 


L Task  requires  much  more  emphasis  In  school 

2.  Training  less  than  adequate  for  task, 
Increase  emphasis. 

3.  T raining  adequate  for  task. 

1 Training  more  than  adequate  for  task, 
reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for 
this  task. 


FREQUENCY  OF  TASK 


L Never  performed.* 

2.  Seldom  performed  or  only  in  emergencies. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  daily. 


•ADEQUACY  scale  may  be  skipped  if  task  Is 
{ never  performed. 


2 3 4 


5 1 


2 3 4 


5 
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SUPERVISOR 

ADEQUACY  OF  SCHOOL  TRAINING 
FOR  THIS  TASK 

L Task  requires  much  more  emphasis  In  schoo 

2.  Training  less  tlian  adequate  lor  task. 
Increase  emphasis. 

3.  Training  adequate  for  task. 

4.  Training  more  than  adeqjate  for  task, 
reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for 
this  task. 


1 2 3 4 5 
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